NeurIPS 2024

Meta Info

Homepage: https://neurips.cc/Conferences/2024

Paper list: https://neurips.cc/virtual/2024/papers.html?filter=titles

Acceptance Rate

• Total: 15671

• Accept: 25.8% (4037)

  • Poster: 23.3% (3650)

  • Spotlight: 2.1% (326)

  • Oral: 0.4% (61)

Papers

Large Language Models (LLMs)

• LLM Inference

  • SGLang: Efficient Execution of Structured Language Model Programs [Paper] [Code] [arXiv]

    • Stanford & UC Berkeley

    • Co-design both the front-end language (programming interface) and the back-end runtime

    • SGLang Primitives

      • Enable the manipulation of prompts and generations

        • gen: call LLM generation

        • select: let the LLM choose the option with the highest probability from a list

        • extend or +=: extend the current prompt

      • Control of parallelism

        • fork: fork the current prompt state

        • join: rejoin the forked prompt states

    • Compilation optimizations

      • Code movement for improving prefix sharing

        • Doesn't strictly preserve the original computation —— aggressive

        • Prompt GPT-4 to re-order graph nodes

    • Runtime

      • RadixAttention

        • Utilize a radix tree (w/ efficient prefix search, reuse, insertion, eviction)

        • LRU eviction policy

      • Cache-aware scheduling → Increase the cache hit rate

        • Key idea: Sort the requests by matched prefix length

  • Efficient LLM Scheduling by Learning to Rank [Paper] [Code]

    • UCSD & THU & Snowflake & UC-Berkeley

    • Insight: it is possible to predict the relative ranks of output lengths in a batch of requests.

    • Develop a scheduler for LLM inference that can approximate the shortest-job-first (SJF) schedule better than existing approaches

• Compound AI Systems

  • Are More LM Calls All You Need? Towards the Scaling Properties of Compound AI Systems [Paper] [Code]

    • Stanford & UC Berkeley & Princeton

    • Systematically study how the number of LM calls affects the performance of two natural inference strategy designs.

      • Vote: Aggregate LM responses via majority voting

      • Filter-Vote: Majority voting after filtering results with an LM

    • Insight

      • More LM calls lead to higher performance on “easy” queries, but lower performance on “hard” queries, and nonmonotone behavior can emerge when a task contains both types of queries.

    • An analytical scaling model to predict the performance of Vote and Filter-Vote systems and find the optimal number of LM calls to make.

Diffusion Models

• Adapter Selection

  • Stylus: Automatic Adapter Selection for Diffusion Models [Paper] [Homepage] [Code]

    • UC Berkeley & CMU & Google DeepMind

    • Problem: how to match the prompt to a set of relevant adapters

    • Stylus

      • Select and automatically compose task-specific adapters based on a prompt's keywords

      • Three-stage approach

        1. Refiner: Leverage visual-language foundational models (VLM) to generate semantic descriptions of adapters then translate them into embeddings

        2. Retriever: Fetch the most relevant adapters over the entirety of the user’s prompt using cosine similarity

        3. Composer: Segment the prompt into tasks from a prompt’s keywords and assign retrieved adapters to tasks

    • StylusDocs

      • An adapter dataset consists of 75K LoRAs (sourced from Civitai) with pre-computed adapter embeddings

• Inference

  • Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference [Paper]

    • HKUST & HKU & Salesforce AI Research & UIUC

    • Develop a general RTK (reverse transition kernel) framework that enables a more balanced subproblem decomposition

    • Propose leveraging two fast sampling algorithms, the Metropolis-Adjusted Langevin Algorithm (MALA) and Underdamped Langevin Dynamics (ULD), for solving these strongly log-concave subproblems

  • Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity [Paper]

    • Stanford

    • Propose to divide the sampling process into $O(1)$ blocks with parallelizable Picard iterations within each block
• Talking Face Video Generation

  • VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time [Paper] [Homepage]

    • MSRA

    • A framework to generate lifelike talking faces with appealing visual affective skills (VAS).

    • A diffusion-based holistic facial dynamics and head movement generation model that works in a face latent space.

    • Support the online generation of 512×512 videos at up to 40 FPS.

• Facial Parts Swapping

  • FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images [Paper] [Code (coming...)]

    • Alibaba

    • Facial parts from different people are assembled into a complete face in latent space within the Mask-based Fusion Module

    • The consolidated feature is dispatched to the Addition-based Injection Module for fusion within the UNet of the diffusion model to create novel characters

Autoregressive Image Generation

• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction [Paper] [Code] [arXiv]

  • PKU & ByteDance

  • Best Paper Award

  • VAR: Visual Autoregressive Modeling

  • Redefine the autoregressive learning on images as coarse-to-fine “next-scale prediction” or “next-resolution prediction”

  • Multi-scale token maps are autoregressively generated from coarse to fine scales (lower to higher resolutions), with parallel token generation within each scale

• Autoregressive Image Generation without Vector Quantization [Paper] [Code] [arXiv]

  • MIT & Google DeepMind & THU

  • Propose to model the per-token probability distribution using a diffusion procedure

  • Define a Diffusion Loss function to model the per-token probability

  • Evaluated across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants

Text-to-Video Generation

• Inference

  • Fast and Memory-Efficient Video Diffusion Using Streamlined Inference [Paper] [Code]

    • NEU

    • Streamlined Inference: Leverage the temporal and spatial properties of video diffusion models

    • Three core components

      • Feature Slicer: Partition input features into sub-features

      • Operator Grouping: Process each sub-feature with a group of consecutive operators

      • Step Rehash: Accelerate inference through skipping unnecessary steps

• Evaluation

  • VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models [Paper] [Homepage] [Code] [Dataset]

    • UTS & ZJU

    • 1.67M unique text-to-video Prompts from real users.

    • 6.69M videos generated by four state-of-the-art diffusion models (Pika, VideoCraft2, Text2Video-Zero, ModelScope).

  • Evaluation of Text-to-Video Generation Models: A Dynamics Perspective [Paper] [Homepage] [Code]

    • UCAS & HIT & Adelaide & Baidu

      • Existing evaluation protocols primarily focus on temporal consistency and content continuity, yet largely ignore the dynamics of video content.

      • DEVIL: An evaluation protocol that centers on the dynamics dimension to evaluate T2V generation models

  • Boosting Text-to-Video Generative Model with MLLMs Feedback [Paper]

    • MSRA

      • Utilize Multimodal Large Language Models (MLLMs) to perform fine-grained video preference annotations → VideoPrefer (13.5K preference annotations)

      • VideoRM: The reward model for text-to-video alignment