📜
Awesome Papers
  • Introduction
  • Paper List
    • Systems for ML
      • Data Processing
      • Deep Learning Training
      • Resource Scheduler
      • Model Serving
      • Large Language Model (LLM)
      • Diffusion Models
      • Deep Learning Recommendation Model (DLRM)
      • Mixture of Experts (MoE)
      • Hyper-Parameter Tuning (HPO)
      • Reinforcement Learning (RL)
      • Deep Learning Compiler
      • Deep Learning Framework
      • Cloud-Edge Collaboration
    • ML for Systems
    • Artificial Intelligence (AI)
      • Diffusion Models
      • Language Models
      • Deep Learning Recommendation Model (DLRM)
    • Hardware Virtualization
      • GPU Sharing
    • Resource Disaggregation
      • GPU Disaggregation
      • Memory Disaggregation
    • Resource Fragmentation
    • Cloud Computing
      • Sky Computing
      • Serverless Computing
      • Spot Instances
    • Remote Direct Memory Access (RDMA)
    • Research Skills
    • Miscellaneous
  • Reading Notes
    • Conference
      • ICML 2025
      • ATC 2025
      • OSDI 2025
      • HotOS 2025
      • MLSys 2025
      • NSDI 2025
      • ASPLOS 2025
      • EuroSys 2025
      • HPCA 2025
      • PPoPP 2025
      • NeurIPS 2024
      • SoCC 2024
      • HotNets 2024
      • SC 2024
      • SOSP 2024
      • VLDB 2024
      • SIGCOMM 2024
      • ICML 2024
      • ATC 2024
      • OSDI 2024
      • ISCA 2024
      • CVPR 2024
      • MLSys 2024
      • ASPLOS 2024
        • SpotServe: Serving generative large language models on preemptible instances
      • EuroSys 2024
        • Orion: Interference-aware, fine-grained GPU sharing for ML applications
      • NSDI 2024
      • NeurIPS 2023
      • SC 2023
        • Interference-aware multiplexing for deep learning in GPU clusters: A middleware approach
      • SoCC 2023
      • SOSP 2023
        • UGache: A unified GPU cache for embedding-based deep learning
      • SIGCOMM 2023
      • HotChips 2023
      • ICML 2023
      • ATC 2023
        • Accelerating Distributed MoE Training and Inference with Lina
        • SmartMoE: Efficiently Training Sparsely-Activated Models ...
        • Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent
      • OSDI 2023
      • HotOS 2023
      • SIGMOD 2023
      • ISCA 2023
      • MLSys 2023
      • EuroSys 2023
      • NSDI 2023
        • Shepherd: Serving DNNs in the wild
        • Understanding RDMA microarchitecture resources for performance isolation
        • Skyplane: Optimizing transfer cost and throughput using cloud-aware overlays
        • Shockwave: Fair and efficient cluster scheduling for dynamic adaptation in machine learning
      • ASPLOS 2023
        • TPP: Transparent page placement for CXL-enabled tiered-memory
        • EVStore: Storage and caching capabilities for scaling embedding tables in deep recommendation system
        • Lucid: A non-intrusive, scalable and interpretable scheduler for deep learning training jobs
      • SC 2022
      • SoCC 2022
        • ESCHER: Expressive scheduling with ephemeral resources
        • Serving unseen deep learning model with near-optimal configurations: A fast adaptive search approach
      • SIGCOMM 2022
        • Multi-resource interleaving for deep learning training
      • ATC 2022
        • PilotFish: Harvesting Free Cycles of Cloud Gaming with Deep Learning Training
        • Memory Harvesting in Multi-GPU Systems with Hierarchical Unified Virtual Memory
        • Whale: Efficient Giant Model Training over Heterogeneous GPUs
        • DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Service...
        • Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing
        • SOTER: Guarding Black-box Inference for General Neural Networks at the Edge
        • Direct access, high-performance memory disaggregation with DirectCXL
      • OSDI 2022
        • Orca: A distributed serving system for transformer-based generative models
        • Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences
        • Looking beyond GPUs for DNN scheduling on multi-tenant clusters
      • IPDPS 2022
        • DGSF: Disaggregated GPUs for serverless functions
      • EuroSys 2022
        • Slashing the disaggregation tax in heterogeneous data centers with FractOS
      • NSDI 2022
      • SoCC 2021
      • ATC 2021
        • Zico: Efficient GPU memory sharing for concurrent DNN training
      • OSDI 2021
        • Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning
      • SOSP 2021
        • HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM
      • EuroSys 2021
        • Take it to the limit: Peak prediction-driven resource overcommitment in datacenters
      • HotOS 2021
        • From cloud computing to sky computing
      • NSDI 2021
      • OSDI 2020
        • A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters
        • HiveD: Sharing a GPU cluster for deep learning with guarantees
      • ATC 2020
        • Serverless in the wild: Characterizing and optimizing the serverless workload
      • EuroSys 2020
      • ASPLOS 2020
      • MLSys 2020
      • SoCC 2020
        • Elastic Parameter Server Load Distribution in Deep Learning Clusters
      • HPDC 2020
        • KubeShare: A framework to manage GPUs as first-class and shared resources in container cloud
      • CLUSTER 2019
      • EuroSys 2019
      • NSDI 2019
      • IWQoS 2019
        • Who limits the resource efficiency of my datacenter: An analysis of Alibaba datacenter traces
      • SIGCOMM 2018
        • Revisiting network support for RDMA
      • OSDI 2018
        • Ray: A distributed framework for emerging AI applications
      • EuroSys 2018
        • Medea: Scheduling of long running applications in shared production clusters
      • ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018
        • GaiaGPU: Sharing GPUs in container clouds
      • SoCC 2017
        • SLAQ: Quality-driven scheduling for distributed machine learning
      • ASPLOS 2017
        • Neurosurgeon: Collaborative intelligence between the cloud and mobile edge
      • NSDI 2017
        • Clipper: A low-latency online prediction serving system
      • CLUSTER 2014
        • Evaluating job packing in warehouse-scale computing
    • Journal
      • IEEE Transactions on Cloud Computing
        • 2021
          • Gemini: Enabling multi-tenant GPU sharing based on kernel burst estimation
      • ACM Computing Surveys
        • 2017
          • GPU virtualization and scheduling methods: A comprehensive survey
      • ACM SIGCOMM Computer Communication Review (CCR)
        • 2021
          • Data-driven Networking Research: models for academic collaboration with industry
        • 2007
          • How to Read a Paper
      • Communications of the ACM
        • 2015
          • Why Google stores billions of lines of code in a single repository
    • Miscellaneous
      • arXiv
        • 2024
          • Efficiently programming large language models using SGLang
        • 2023
          • HexGen: Generative inference of foundation model over heterogeneous decentralized environment
          • High-throughput generative inference of large language models with a single GPU
        • 2022
          • DisaggRec: Architecting disaggregated systems for large-scale personalized recommendation
          • A case for disaggregation of ML data processing
          • Singularity: Planet-scale, preemptive and elastic scheduling of AI workloads
          • Aryl: An elastic cluster scheduler for deep learning
        • 2016
          • Wide & deep learning for recommender systems
          • Training deep nets with sublinear memory cost
      • MSR Technical Report
        • 2011
          • Heuristics for vector bin packing
  • About Myself
    • Academic Profile
    • Personal Blog (in Chinese)
Powered by GitBook
On this page
  • Meta Info
  • Papers
  • Large Language Models (LLMs)
  • Mixture-of-Experts (MoEs)
  • Diffusion Models
  • Deep Learning Recommendation Models (DLRMs)
  • ML Compilation
  • Quantization
  • Model Adaptation
  • Cloud Configuration Generation
  • Acronyms

Was this helpful?

Edit on GitHub
  1. Reading Notes
  2. Conference

MLSys 2024

Last updated 4 months ago

Was this helpful?

Meta Info

Homepage:

Paper list:

Papers

Large Language Models (LLMs)

  • LoRA serving

    • S-LoRA: Serving Thousands of Concurrent LoRA Adapters [] [] []

      • UC Berkeley

        • A system to serve many LoRA adapters

        • Store all adapters in the main memory and fetch the adapters used by the currently running queries to the GPU memory

        • Unified Paging — a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths

        • Employ a tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation

        • Built on top of

    • Punica: Multi-Tenant LoRA Serving [] []

      • UW & Duke

        • A system to serve multiple LoRA models in a shared GPU cluster

        • A CUDA kernel — Segmented Gather Matrix-Vector Multiplication (SGMV)

          • Batch GPU operations for concurrent execution of different LoRA models

          • A GPU only needs to store a single copy of the pre-trained model

        • A request scheduling mechanism to consolidate multi-tenant LoRA serving workloads

          • Route the new request to a small set of active GPUs

          • Allocate additional GPU resources when the existing GPUs are fully utilized

          • Periodically migrate existing requests for consolidation

  • LLM inference

    • Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference [] []

      • UBC & d-Matrix

    • Prompt Cache: Modular Attention Reuse for Low-Latency Inference []

      • Yale & Google

    • HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices []

      • NUS

    • Vidur: A Large-scale Simulation Framework for LLM Inference [] []

      • GaTech & MSR India

    • FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics []

      • THU & Infinigence-AI

  • LLM fine-tuning

    • Fine-Tuning Language Models Using Formal Methods Feedback: A Use Case in Autonomous Systems []

      • UT-Austin

  • LLM for data manipulation

    • UniDM: A Unified Framework for Data Manipulation with Large Language Models []

      • Alibaba & USTC

Mixture-of-Experts (MoEs)

  • MoE training

      • HKU & AWS & Boson AI

  • MoE inference

      • Institute of Science and Technology Austria

    • SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

Diffusion Models

    • HKU & AWS

Deep Learning Recommendation Models (DLRMs)

    • Meta AI

ML Compilation

    • CMU

    • Perform hybrid static+dynamic compiler optimizations and end-to-end tensor code generation

Quantization

  • FP8

      • Intel

  • LLM

      • MIT

      • Best Paper Award

      • UW

      • UT-Texas & Oxford & Eindhoven University of Technology & Lawrence Livermore National Laboratory & CMU

  • ML training

      • AMD

Model Adaptation

Cloud Configuration Generation

    • Alibaba Cloud & UMich & UCLA & UC Merced

Acronyms

  • ML: Machine Learning

  • LLM: Large Language Model

  • LoRA: Low-Rank Adaptation

  • MoE: Mixture-of-Experts

Lancet: Accelerating Mixture-of-Experts Training by Overlapping Weight Gradient Computation and All-to-All Communication []

QMoE: Sub-1-Bit Compression of Trillion Parameter Models [] []

DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines []

Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large Scale Recommendation []

ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time []

Efficient Post-training Quantization with FP8 Formats []

AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration [] []

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving [] [] [] []

Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache [] []

JIT-Q: Just-in-time Quantization with Processing-In-Memory for Efficient ML Training [] []

FLASH: Fast Model Adaptation in ML-Centric Cloud Platforms [Paper] [] []

CloudEval-YAML: A Practical Benchmark for Cloud Native YAML Configuration Generation [] [] [] []

https://mlsys.org/Conferences/2024
https://mlsys.org/Conferences/2024/AcceptedPapers
Paper
arXiv
Code
LightLLM
arXiv
Code
Paper
Code
Paper
Paper
Paper
Code
Paper
Paper
Paper
Paper
Paper
Code
Paper
Paper
Paper
Paper
Paper
Code
Paper
Code
Slides
Poster
Paper
Code
Paper
Slides
Code
Slides
Paper
Homepage
Code
Benchmark