MLSys 2024

Meta Info

Homepage: https://mlsys.org/Conferences/2024

Paper list: https://mlsys.org/Conferences/2024/AcceptedPapers

Papers

Large Language Models (LLMs)

• LoRA serving

  • S-LoRA: Serving Thousands of Concurrent LoRA Adapters [Paper] [arXiv] [Code]

    • UC Berkeley

      • A system to serve many LoRA adapters

      • Store all adapters in the main memory and fetch the adapters used by the currently running queries to the GPU memory

      • Unified Paging — a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths

      • Employ a tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation

      • Built on top of LightLLM

  • Punica: Multi-Tenant LoRA Serving [arXiv] [Code]

    • UW & Duke

      • A system to serve multiple LoRA models in a shared GPU cluster

      • A CUDA kernel — Segmented Gather Matrix-Vector Multiplication (SGMV)

        • Batch GPU operations for concurrent execution of different LoRA models

        • A GPU only needs to store a single copy of the pre-trained model

      • A request scheduling mechanism to consolidate multi-tenant LoRA serving workloads

        • Route the new request to a small set of active GPUs

        • Allocate additional GPU resources when the existing GPUs are fully utilized

        • Periodically migrate existing requests for consolidation

• LLM inference

  • Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference [Paper] [Code]

    • UBC & d-Matrix

  • Prompt Cache: Modular Attention Reuse for Low-Latency Inference [Paper]

    • Yale & Google

  • HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices [Paper]

    • NUS

  • Vidur: A Large-scale Simulation Framework for LLM Inference [Paper] [Code]

    • GaTech & MSR India

  • FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics [Paper]

    • THU & Infinigence-AI

• LLM fine-tuning

  • Fine-Tuning Language Models Using Formal Methods Feedback: A Use Case in Autonomous Systems [Paper]

    • UT-Austin

• LLM for data manipulation

  • UniDM: A Unified Framework for Data Manipulation with Large Language Models [Paper]

    • Alibaba & USTC

Mixture-of-Experts (MoEs)

• MoE training

  • Lancet: Accelerating Mixture-of-Experts Training by Overlapping Weight Gradient Computation and All-to-All Communication [Paper]

    • HKU & AWS & Boson AI

• MoE inference

  • QMoE: Sub-1-Bit Compression of Trillion Parameter Models [Paper] [Code]

    • Institute of Science and Technology Austria

  • SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

Diffusion Models

• DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines [Paper]

  • HKU & AWS

Deep Learning Recommendation Models (DLRMs)

• Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large Scale Recommendation [Paper]

  • Meta AI

ML Compilation

• ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time [Paper]

  • CMU

  • Perform hybrid static+dynamic compiler optimizations and end-to-end tensor code generation

Quantization

• FP8

  • Efficient Post-training Quantization with FP8 Formats [Paper]

    • Intel

• LLM

  • AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration [Paper] [Code]

    • MIT

    • Best Paper Award

  • Atom: Low-bit Quantization for Efficient and Accurate LLM Serving [Paper] [Code] [Slides] [Poster]

    • UW

  • Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache [Paper] [Code]

    • UT-Texas & Oxford & Eindhoven University of Technology & Lawrence Livermore National Laboratory & CMU

• ML training

  • JIT-Q: Just-in-time Quantization with Processing-In-Memory for Efficient ML Training [Paper] [Slides]

    • AMD

Model Adaptation

• FLASH: Fast Model Adaptation in ML-Centric Cloud Platforms [Paper] [Code] [Slides]

Cloud Configuration Generation

• CloudEval-YAML: A Practical Benchmark for Cloud Native YAML Configuration Generation [Paper] [Homepage] [Code] [Benchmark]

  • Alibaba Cloud & UMich & UCLA & UC Merced

Acronyms

• ML: Machine Learning

• LLM: Large Language Model

• LoRA: Low-Rank Adaptation

• MoE: Mixture-of-Experts