# Introduction

> “Think different.” —— Apple Inc.

Hi all, thank you for visiting here ;-)

I'm a senior engineer at Alibaba, working on AI infrastructure. Prior to that, I earned my Ph.D. degree from HKUST.

This space hosts my personal notes on academic papers. My interests span systems (e.g., OSDI, SOSP, NSDI, ATC, EuroSys, SoCC, ASPLOS, MLSys, CAIS), machine learning (e.g., ICML, ICLR, NeurIPS), and other funny stuff.

All views expressed here are my own.

## Welcome

1. I'll aperiodically organize some interesting papers that I have read during my leisure and upload their notes to the [GitHub repository](https://github.com/mental2008/awesome-papers), which can be read easily on the [website](https://paper.lingyunyang.com/).
2. I've organized a list of papers on several topics (e.g., systems for ML) and also categorized the reading notes by conference/journal dates.
3. If you find a paper that interests you in my reading notes, feel free to [create an issue](https://github.com/mental2008/awesome-papers/issues/new) and chat with me.
4. Even if the paper you are interested in is not in my notes, you can also [create an issue](https://github.com/mental2008/awesome-papers/issues/new) to let me know. I will read it and give you my thoughts if I have time.
5. Welcome to [submit a pull request](https://github.com/mental2008/awesome-papers/pulls) if you find any typos.

## Changelogs

* 07/2026: Organize the paper lists of [On-Policy Distillation](/paper-list/artificial-intelligence/opd) and [Continual Pre-training](/paper-list/artificial-intelligence/cpt); organize the papers of [MLSys 2026](/reading-notes/conference/mlsys-2026) and [OSDI 2026](/reading-notes/conference/osdi-2026); update the paper lists of [Systems for LLMs](/paper-list/systems-for-ml/llm) and [Systems for Diffusion Models](/paper-list/systems-for-ml/diffusion-models).
* 06/2026: Organize the papers of [CAIS 2026](/reading-notes/conference/cais-2026).
* 04/2026: Organize the papers of [ASPLOS 2026](/reading-notes/conference/asplos-2026), [EuroSys 2026](/reading-notes/conference/eurosys-2026), [NSDI 2026](/reading-notes/conference/nsdi-2026); create the paper list of [Kernel Generation](/paper-list/ml-for-systems/kernel-generation).
* 01/2026: Organize the papers of [PPoPP 2026](/reading-notes/conference/ppopp-2026), [HPCA 2026](/reading-notes/conference/hpca-2026).
* 10/2025: Organize the papers of [OSDI 2025](/reading-notes/conference/osdi-2025), [ATC 2025](/reading-notes/conference/atc-2025), [SOSP 2025](/reading-notes/conference/sosp-2025).
* 09/2025: Organize the papers of [SIGCOMM 2025](/reading-notes/conference/sigcomm-2025).
* 06/2025: Organize the papers of [ISCA 2025](/reading-notes/conference/isca-2025), [SIGMETRICS 2025](/reading-notes/conference/sigmetrics-2025).
* 05/2025: Organize the papers of [HotOS 2025](/reading-notes/conference/hotos-2025); fix the paper links of [MLSys 2025](/reading-notes/conference/mlsys-2025).
* 03/2025: Organize the papers of [NSDI 2025](/reading-notes/conference/nsdi-2025), [EuroSys 2025](/reading-notes/conference/eurosys-2025) (Fall cycle), [ASPLOS 2025](/reading-notes/conference/asplos-2025), [MLSys 2025](/reading-notes/conference/mlsys-2025); update the paper list of [systems for LLMs](/paper-list/systems-for-ml/llm).
* 02/2025: Organize the papers of [HPCA 2025](/reading-notes/conference/hpca-2025), [PPoPP 2025](/reading-notes/conference/ppopp-2025).
* 01/2025: Update the paper list of [research skills](/paper-list/research-skills) and [systems for diffusion models](/paper-list/systems-for-ml/diffusion-models); organize the papers of [HotNets 2024](/reading-notes/conference/hotnets-2024), [MLSys 2024](/reading-notes/conference/mlsys-2024), [NeurIPS 2024](/reading-notes/conference/neurips-2024).
* 12/2024: Briefly organize the papers of [EuroSys 2025](/reading-notes/conference/eurosys-2025) (only Spring cycle); organize the papers of [SoCC 2024](/reading-notes/conference/socc-2024), [SC 2024](/reading-notes/conference/sc-2024); update the reading notes of [SOSP 2024](/reading-notes/conference/sosp-2024).
* 09/2024: Organize the papers of [SOSP 2024](/reading-notes/conference/sosp-2024).
* 08/2024: Organize the papers of [VLDB 2024](/reading-notes/conference/vldb-2024); update the reading notes of [SIGCOMM 2024](/reading-notes/conference/sigcomm-2024); create several new paper lists of [diffusion models](/paper-list/artificial-intelligence/diffusion-models), [language models](/paper-list/artificial-intelligence/language-models), and [deep learning recommendation models](/paper-list/artificial-intelligence/dlrm).
* 07/2024: Organize the papers of [SIGCOMM 2024](/reading-notes/conference/sigcomm-2024), [ICML 2024](/reading-notes/conference/icml-2024), [ATC 2024](/reading-notes/conference/atc-2024), [OSDI 2024](/reading-notes/conference/osdi-2024), [NSDI 2024](/reading-notes/conference/nsdi-2024), [CVPR 2024](/reading-notes/conference/cvpr-2024), [ISCA 2024](/reading-notes/conference/isca-2024); create a new paper list of [systems for diffusion models](/paper-list/systems-for-ml/diffusion-models); update the paper list of [systems for LLMs](/paper-list/systems-for-ml/llm), [systems for DLRMs](/paper-list/systems-for-ml/dlrm), and [resource scheduler](/paper-list/systems-for-ml/resource-scheduler).

## Epilogue

If it helps, you could star the [GitHub repository](https://github.com/mental2008/awesome-papers).

For a better experience, please check out the [website](https://paper.lingyunyang.com/).

## License

Released under the [MIT License](https://github.com/mental2008/awesome-papers/tree/develop/LICENSE/README.md).

Copyright © 2021—2026 [Lingyun Yang](https://github.com/mental2008).


# Systems for ML

## ML Lifecycle

* [Data Processing](/paper-list/systems-for-ml/data-processing)
* [Deep Learning Training](/paper-list/systems-for-ml/deep-learning-training)
* [Resource Scheduler](/paper-list/systems-for-ml/resource-scheduler)
* [Model Serving](/paper-list/systems-for-ml/model-serving)
* [Deep Learning Compiler](/paper-list/systems-for-ml/deep-learning-compiler)
* [Deep Learning Framework](/paper-list/systems-for-ml/deep-learning-framework)
* [Cloud-Edge Collaboration](/paper-list/systems-for-ml/cloud-edge-collaboration)

## Various Workloads

* [Large Language Model (LLM)](/paper-list/systems-for-ml/llm)
* [Diffusion Models](/paper-list/systems-for-ml/diffusion-models)
* [Deep Learning Recommendation Model (DLRM)](/paper-list/systems-for-ml/dlrm)
* [Mixture of Experts (MoE)](/paper-list/systems-for-ml/moe)
* [Hyper-Parameter Tuning (HPO)](/paper-list/systems-for-ml/hpo)
* [Reinforcement Learning (RL)](/paper-list/systems-for-ml/rl)


# Data Processing

* Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement ([ATC 2024](/reading-notes/conference/atc-2024)) \[[Paper](https://www.usenix.org/conference/atc24/presentation/graur)] \[[Code](https://github.com/eth-easl/pecan-experiments)]
  * ETH & Google
* Disaggregating ML Input Data Processing at Scale ([SoCC 2023](/reading-notes/conference/socc-2023))
  * Google & ETH
* GoldMiner: Elastic Scaling of Training Data Pre-Processing Pipelines for Deep Learning (SIGMOD 2023) \[[Paper](https://dl.acm.org/doi/10.1145/3589773)]
  * Alibaba & PKU
* A case for disaggregation of ML data processing (arXiv 2210.14826) \[[Paper](https://arxiv.org/abs/2210.14826)]
  * Google & ETH
  * tf.data service: *Disaggregate* data preprocessing from ML computation.
* Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training (ISCA 2022) \[[Paper](https://dl.acm.org/doi/10.1145/3470496.3533044)]
  * Meta
  * DSI: Data storage and ingestion
  * Industry track
  * Meta's data storage and ingestion pipeline


# Deep Learning Training

## Elastic Training

* EasyScale: Elastic Training with Consistent Accuracy and Improved Utilization on GPUs ([SC 2023](/reading-notes/conference/sc-2023)) \[[Paper](https://doi.org/10.1145/3581784.3607054)] \[[Code](https://github.com/sUntvoOk/EasyScale_info_for_SC23)]
  * BUAA & Alibaba

## Parallelism

* Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency ([SC 2023](/reading-notes/conference/sc-2023)) \[[Paper](https://doi.org/10.1145/3581784.3607073)] \[[Code](https://github.com/MaruyamaAya/Wpipe)]
  * NUS
* Supporting Very Large Models using Automatic Dataflow Graph Partitioning ([EuroSys 2019](/reading-notes/conference/eurosys-2019)) \[[Paper](https://doi.org/10.1145/3302424.3303953)]
  * NYU
  * Tofu: *Automatic partition* a dataflow graph of fine-grained tensor operations.
* One weird trick for parallelizing convolutional neural networks (arXiv 1404.599) \[[Paper](https://arxiv.org/abs/1404.5997)]
  * Google
  * *Data parallelism* for *convolutional layers*; *model parallelism* for *fully-connected layers*.

## Optimizing Network Communication

* A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Personal Notes](https://github.com/mental2008/awesome-papers/blob/develop/reading-notes/conference/osdi-2020/a-unified-architecture-for-accelerating-distributed-dnn-training-in-heterogeneous-gpu-cpu-clusters.md)] \[[Paper](https://www.usenix.org/conference/osdi20/presentation/jiang)] \[[Code](https://github.com/bytedance/byteps)]
  * THU & ByteDance
  * BytePS: Communication framework
  * Leverage spare CPU and bandwidth resources
  * Consider network topology

## Reduce GPU Memory Footprints

### GPU Sharing

* Zico: Efficient GPU Memory Sharing for Concurrent DNN Training ([ATC 2021](/reading-notes/conference/atc-2021)) \[[Personal Notes](/reading-notes/conference/atc-2021/zico)] \[[Paper](https://www.usenix.org/conference/atc21/presentation/lim)]
  * UNIST & Ajou & Alibaba & KAIST
  * Reduce the *overall* GPU consumption for *co-located* DNN training jobs
  * Utilize NVIDIA MPS
* Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications ([MLSys 2020](/reading-notes/conference/mlsys-2020)) \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2020/hash/d9cd83bc91b8c36a0c7c0fcca59228f2-Abstract.html)] \[[Code](https://github.com/symbioticlab/salus)]
  * UMich SymbioticLab
  * Fine-grained GPU sharing; customized TensorFlow.
* Gandiva: Introspective Cluster Scheduling for Deep Learning ([OSDI 2018](/reading-notes/conference/osdi-2018)) \[[Paper](https://www.usenix.org/conference/osdi18/presentation/xiao)]
  * MSRA
  * Time slicing; suspend and resume; mini-batch granularity.

### Tensor Swapping / Recomputation

* SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping ([ASPLOS 2020](/reading-notes/conference/asplos-2020)) \[[Paper](https://dl.acm.org/doi/10.1145/3373376.3378530)]
  * NYU
  * Tensor swapping
  * Consider both GPU memory allocation and operator scheduling
* Capuchin: Tensor-based GPU Memory Management for Deep Learning ([ASPLOS 2020](/reading-notes/conference/asplos-2020)) \[[Paper](https://dl.acm.org/doi/10.1145/3373376.3378505)]
  * HUST & MSRA & USC
  * Combination of tensor swapping and recomputation.
* Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization ([MLSys 2020](/reading-notes/conference/mlsys-2020)) \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2020/hash/0b816ae8f06f8dd3543dc3d9ef196cab-Abstract.html)] \[[Code](https://github.com/parasj/checkmate)]
  * UC Berkeley
  * Define tensor recomputation as an optimization problem.
* SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks (PPoPP 2018) \[[Paper](https://dl.acm.org/doi/10.1145/3200691.3178491)]
  * Brown & UESTC & Los Alamos National Laboratory & Pacific Northwest National Laboratory & MIT
  * Cost-aware recomputation
  * Remove the convolutional layer tensor with low computational overhead
* vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design (MICRO 2016) \[[Paper](https://dl.acm.org/doi/10.5555/3195638.3195660)]
  * NVIDIA
  * Predictively swap tensors to overlap the CPU-GPU communication time.
* Training Deep Nets with Sublinear Memory Cost (arXiv 1604.06174) \[[Personal Notes](https://github.com/mental2008/awesome-papers/blob/develop/Miscellaneous/arXiv-2016/training-deep-nets-with-sublinear-memory-cost.md)] \[[Paper](https://arxiv.org/abs/1604.06174)] \[[Code](https://github.com/dmlc/mxnet-memonger)]
  * UW & Dato Inc. & MIT
  * Memory Monger
  * Sublinear memory cost; trade computation for memory.

### Compression

* Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training (ISCA 2020) \[[Paper](https://dl.acm.org/doi/abs/10.1109/ISCA45697.2020.00092)]
  * UofT
  * LSTM RNN training
* Gist: Efficient Data Encoding for Deep Neural Network Training (ISCA 2018) \[[Paper](https://www.microsoft.com/en-us/research/uploads/prod/2018/04/fiddle-gist-isca18.pdf)]
  * MSR & UMich & UofT
  * Data encoding


# Resource Scheduler

{% hint style="info" %}
I am actively maintaining this list.
{% endhint %}

## Scheduling for DL Training Workloads

* CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/rajasekaran)]
  * MIT & UT-Austin
  * Consider the communication pattern of different jobs while placing them on network links.
* Blox: A Modular Toolkit for Deep Learning Schedulers ([EuroSys 2024](/reading-notes/conference/eurosys-2024)) \[[arXiv](https://arxiv.org/abs/2312.12621)] \[[Code](https://github.com/msr-fiddle/blox)]
  * UW-Madison & MSR
* Interference-aware Multiplexing for Deep Learning in GPU Clusters: A Middleware Approach ([SC 2023](/reading-notes/conference/sc-2023)) \[[Personal Notes](/reading-notes/conference/sc-2023/iadeep)] \[[Paper](https://doi.org/10.1145/3581784.3607060)] \[[Code](https://github.com/buzy-coder/IADeep)]
  * UMacau & SIAT, CAS
  * IADeep — a cluster scheduler to co-locate DL training tasks
  * Tune training configurations (e.g., batch size) across all co-located tasks; choose appropriate tasks to multiplex on a GPU device; consider PCIe bandwidth
* Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613175)]
  * CMU & Cornell & Petuum Inc.
* Lyra: Elastic Scheduling for Deep Learning Clusters ([EuroSys 2023](/reading-notes/conference/eurosys-2023)) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2022/aryl)] \[[Paper](https://dl.acm.org/doi/10.1145/3552326.3587445)] \[[arXiv](https://arxiv.org/abs/2202.07896)]
  * ByteDance & CityU & CUHK
  * Loan idle inference GPU servers for elastic training jobs.
* Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Personal Notes](/reading-notes/conference/nsdi-2023/shockwave)] \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/zheng)] \[[Code](https://github.com/uw-mad-dash/shockwave)]
  * UW-Madison & UT-Austin
  * Elastic resource requirements; extend market theory.
* Lucid: A Non-intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ([ASPLOS 2023](/reading-notes/conference/asplos-2023)) \[[Personal Notes](/reading-notes/conference/asplos-2023/lucid)] \[[Paper](https://dl.acm.org/doi/10.1145/3575693.3575705)] \[[Code](https://github.com/S-Lab-System-Group/Lucid)]
  * NTU & Shanghai AI Lab & SenseTime
  * Scheduling interpretability
* Multi-Resource Interleaving for Deep Learning Training ([SIGCOMM 2022](/reading-notes/conference/sigcomm-2022)) \[[Personal Notes](/reading-notes/conference/sigcomm-2022/multi-resource-interleaving-for-deep-learning-training)] \[[Paper](https://dl.acm.org/doi/10.1145/3544216.3544224)] \[[Code](https://github.com/Rivendile/Muri)]
  * PKU & ByteDance
  * Muri: Pack jobs along multiple resource types in the time dimension
  * Integrate with PyTorch
* Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads (arXiv 2202.07848) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2022/singularity)] \[[Paper](https://arxiv.org/abs/2202.07848)]
  * Microsoft
  * Live GPU job migration
* Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Personal Notes](/reading-notes/conference/osdi-2022/synergy)] \[[Paper](https://www.usenix.org/conference/osdi22/presentation/mohan)] \[[Code](https://github.com/msr-fiddle/synergy)]
  * MSR & UT-Austin & VMware Research
  * Consider the allocation of CPU and memory resources.
* Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning ([OSDI 2021](/reading-notes/conference/osdi-2021)) \[[Personal Notes](/reading-notes/conference/osdi-2021/pollux)] \[[Paper](https://www.usenix.org/conference/osdi21/presentation/qiao)] \[[Code](https://github.com/petuum/adaptdl)]
  * Petuum & CMU
  * **Best Paper Award**
  * Co-adaptively allocates resources (number of GPUs) and tunes the hyperparameters (batch size and learning rate) for all DL training jobs.
* MAPA: Multi-Accelerator Pattern Allocation Policy for Multi-Tenant GPU Servers (SC 2021) \[[Paper](https://dl.acm.org/doi/10.1145/3458817.3480853)] \[[Code](https://github.com/socal-ucr/MAPA)]
  * UC Riverside & Pacific Northwest National Lab & USydney
  * Consider multi-GPU accelerator topologies such as single/double NVLink.
* Astraea: A Fair Deep Learning Scheduler for Multi-Tenant GPU Clusters (TPDS 2021) \[[Paper](https://ieeexplore.ieee.org/abstract/document/9655467)]
  * PKU & NTU & SenseTime
  * Long-term GPU-time fairness
* AntMan: Dynamic Scaling on GPU Clusters for Deep Learning ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Paper](https://www.usenix.org/conference/osdi20/presentation/xiao)] \[[Code](https://github.com/alibaba/GPU-scheduler-for-deep-learning)]
  * Alibaba
  * Co-locate resource-guarantee and best-effort jobs.
* HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Personal Notes](/reading-notes/conference/osdi-2020/hived)] \[[Paper](https://www.usenix.org/conference/osdi20/presentation/zhao-hanyu)] \[[Code](https://github.com/microsoft/hivedscheduler)]
  * MSRA
  * Virtual private clusters; resource isolation and management for multi-tenant clusters.
* Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Paper](https://www.usenix.org/conference/osdi20/presentation/narayanan-deepak)] \[[Code](https://github.com/stanford-futuredata/gavel)]
  * MSR & Stanford
  * Gavel: Consider performance heterogeneity across multiple accelerator types.
* Themis: Fair and Efficient GPU Cluster Scheduling (NSDI 2020) \[[Paper](https://www.usenix.org/conference/nsdi20/presentation/mahajan)]
  * UW-Madison & MSR
  * Long-term fairness
* AlloX: Compute Allocation in Hybrid Clusters ([EuroSys 2020](/reading-notes/conference/eurosys-2020)) \[[Paper](https://doi.org/10.1145/3342195.3387547)] \[[Code](https://github.com/lenhattan86/allox)]
  * Stony Brook University & SUNY Korea & UMich
  * CPU-GPU hybrid clusters; min-cost bipartite matching.
* Balancing Efficiency and Fairness in Heterogeneous GPU Clusters for Deep Learning ([EuroSys 2020](/reading-notes/conference/eurosys-2020)) \[[Paper](https://dl.acm.org/doi/10.1145/3342195.3387555)]
  * MSR India
  * $$\text{Gandiva}\_\text{Fair}$$: Achieve efficiency and fairness despite cluster heterogeneity
* Tiresias: A GPU Cluster Manager for Distributed Deep Learning ([NSDI 2019](/reading-notes/conference/nsdi-2019)) \[[Paper](https://www.usenix.org/conference/nsdi19/presentation/gu)] \[[Code](https://github.com/SymbioticLab/Tiresias)]
  * UMich SymbioticLab
  * Relax consolidated placement constraint
* Gandiva: Introspective Cluster Scheduling for Deep Learning ([OSDI 2018](/reading-notes/conference/osdi-2018)) \[[Paper](https://www.usenix.org/conference/osdi18/presentation/xiao)]
  * MSRA
  * Hyper-parameter tuning jobs; job packing; migration; grow-shrink; time-slicing.
* Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters (EuroSys 2018) \[[Paper](https://dl.acm.org/doi/10.1145/3190508.3190517)] \[[Code](https://github.com/pengyanghua/optimus)]
  * HKU & ByteDance
  * Minimize JCT based on online resource-performance models.
* Topology-Aware GPU Scheduling for Learning Workloads in Cloud Environments (SC 2017) \[[Paper](https://dl.acm.org/doi/10.1145/3126908.3126933)] \[[Code](https://github.com/HiEST/gpu-topo-aware)]
  * Barcelona Supercomputing Center & IBM Watson Research Center
  * Consider multiple link technologies such as PCI-e and NVLink.

## Scheduling for General ML Training Workloads

* SLAQ: Quality-Driven Scheduling for Distributed Machine Learning ([SoCC 2017](/reading-notes/conference/socc-2017)) \[[Personal Notes](/reading-notes/conference/socc-2017/slaq)] \[[Paper](https://dl.acm.org/doi/10.1145/3127479.3127490)]
  * Princeton
  * Fine-grained job-level scheduler
  * Leverage the iterative nature of general ML training algorithms

## Trace Analysis

* MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters ([NSDI 2022](/reading-notes/conference/nsdi-2022)) \[[Paper](https://www.usenix.org/conference/nsdi22/presentation/weng)] \[[Trace](https://github.com/alibaba/clusterdata/tree/master/cluster-trace-gpu-v2020)]
  * HKUSt & Alibaba
  * GPU sharing traces
* Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters (SC 2021) \[[Paper](https://dl.acm.org/doi/10.1145/3458817.3476223)] \[[Trace](https://github.com/S-Lab-System-Group/HeliosData)]
  * NTU & SenseTime
* Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (ATC 2019) \[[Paper](https://www.usenix.org/conference/atc19/presentation/jeon)] \[[Trace](https://github.com/msr-fiddle/philly-traces)]
  * MSR
* Characterizing Deep Learning Training Workloads on Alibaba-PAI (IISWC 2019) \[[Paper](https://ieeexplore.ieee.org/document/9042047)]
  * Alibaba PAI

## Survey

* Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision (arXiv 2205.11913) \[[Paper](https://arxiv.org/abs/2205.11913)] \[[Paper List](https://github.com/S-Lab-System-Group/Awesome-DL-Scheduling-Papers)]
  * NTU & PKU & SenseTime

## Acronyms

* DL: Deep Learning
* ML: Machine Learning


# Model Serving

{% hint style="warning" %}
Large language models (LLMs) are *hot* and *diverse* compared to conventional models. Therefore, I have classified the related works for LLMs in [another paper list](/paper-list/systems-for-ml/llm).
{% endhint %}

{% hint style="info" %}
I am actively maintaining this list.
{% endhint %}

## Model Serving Systems

* Usher: Holistic Interference Avoidance for Resource Optimized ML Inference ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/shubha)] \[[Code](https://github.com/ss7krd/Usher)]
  * UVA & GaTech
* Paella: Low-latency Model Serving with Software-defined GPU Scheduling ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613163)]
  * UPenn & DBOS, Inc.
* Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Personal Notes](/reading-notes/conference/osdi-2022/reef)] \[[Paper](https://www.usenix.org/conference/osdi22/presentation/han)] \[[Code](https://github.com/SJTU-IPADS/reef)] \[[Benchmark](https://github.com/SJTU-IPADS/disb)] \[[Artifact](https://github.com/SJTU-IPADS/reef-artifacts/tree/osdi22-ae)]
  * SJTU
  * REEF: GPU kernel preemption; dynamic kernel padding.
* INFaaS: Automated Model-less Inference Serving ([ATC 2021](/reading-notes/conference/atc-2021)) \[[Paper](https://www.usenix.org/conference/atc21/presentation/romero)] \[[Code](https://github.com/stanford-mast/INFaaS)]
  * Stanford
  * **Best Paper**
  * Consider *model-variants*
* Clipper: A Low-Latency Online Prediction Serving System ([NSDI 2017](/reading-notes/conference/nsdi-2017)) \[[Personal Notes](/reading-notes/conference/nsdi-2017/clipper)] \[[Paper](https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw)] \[[Code](https://github.com/ucbrise/clipper)]
  * UC Berkeley
  * Caching, batching, adaptive model selection.
* TensorFlow-Serving: Flexible, High-Performance ML Serving (NIPS 2017 Workshop on ML Systems) \[[Paper](https://arxiv.org/abs/1712.06139)]
  * Google

## Auto-Configuration for Model Serving

* Serving Unseen Deep Learning Models with Near-Optimal Configurations: a Fast Adaptive Search Approach ([SoCC 2022](/reading-notes/conference/socc-2022)) \[[Personal Notes](/reading-notes/conference/socc-2022/falcon)] \[[Paper](https://dl.acm.org/doi/10.1145/3542929.3563485)] \[[Code](https://github.com/dos-lab/Falcon)]
  * ISCAS
  * Characterize a DL model by its key operators.
* Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving ([SoCC 2021](/reading-notes/conference/socc-2021)) \[[Paper](https://dl.acm.org/doi/10.1145/3472883.3486987)] \[[Code](https://github.com/kubedl-io/morphling)]
  * HKUST & Alibaba
  * Meta learning; bayesian optimization; Kubernetes.

## Survey

* A Survey of Multi-Tenant Deep Learning Inference on GPU (MLSys 2022 Workshop on Cloud Intelligence / AIOps) \[[Paper](https://arxiv.org/abs/2203.09040)]
  * George Mason & Microsoft & Maryland
* A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities (arXiv 2111.14247) \[[Paper](https://arxiv.org/abs/2111.14247)]
  * George Mason & Microsoft & Pittsburgh & Maryland


# Large Language Model (LLM)

{% hint style="info" %}
I am actively maintaining this list.
{% endhint %}

## LLM Training

### Hybrid Parallelism

* Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Parallelism ([ATC 2024](/reading-notes/conference/atc-2024)) \[[Paper](https://www.usenix.org/conference/atc24/presentation/yuan)] \[[Code](https://github.com/kwai/Megatron-Kwai/tree/atc24ae/examples/atc24)]
  * Kuaishou
* Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Paper](https://www.usenix.org/conference/osdi22/presentation/zheng-lianmin)] \[[Code](https://github.com/alpa-projects/alpa)] \[[Docs](https://alpa.ai/)]
  * UC Berkeley & AWS & Google & SJTU & CMU & Duke
  * Generalize the search through *parallelism strategies*.

### RL Post-Training

{% hint style="info" %}
**Reading map.** These works differ along six axes:

* **Disaggregated RL systems**: split rollout, inference, environment, reward, and training stages across best-fit resources.

* **Workflow scheduling and resource reallocation**: reshape RL pipelines or dynamically move compute, memory, communication, and parameters across roles.

* **Rollout latency and long-tail mitigation**: predict, batch, reuse, or otherwise reduce long-tail rollout work.

* **Speculative decoding for RL**: adapt draft-and-verify generation to RL training constraints such as drafter staleness and rollout distribution shift.

* **Heterogeneous environments**: make RL training efficient across mixed GPU generations and hardware capabilities.

* **Fault tolerance**: isolate and recover failures across trainer, rollout, and control-plane roles.
  {% endhint %}

* RollArt: Disaggregated Multi-Task Agentic RL Training at Scale ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/gao)]
  * HKUST & Alibaba & Tongyi Lab
  * Axis: **Disaggregated RL systems**
  * Maps agentic RL pipeline stages to best-fit hardware and coordinates prefill, decoding, environment, and reward workloads.

* Weave: Efficient Co-Scheduling for Disaggregated RL Post-Training ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wu-tianyuan)]
  * HKUST & UIUC & Alibaba
  * Axis: **Disaggregated RL systems**; **Workflow scheduling and resource reallocation**
  * Reclaims dependency bubbles between rollout and training phases in disaggregated RL post-training.

* RLinf: Flexible and Efficient Large-Scale Reinforcement Learning via Macro-to-Micro Flow Transformation ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/yu-chao)]
  * THU & Infinigence-AI & PKU & UC Berkeley & Zhongguancun Academy & BUAA & SJTU
  * Axis: **Workflow scheduling and resource reallocation**
  * Transforms RL workflows from macro pipelines into micro flows to improve scheduling flexibility and hardware utilization.

* DynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wang-yuanqing)]
  * PKU & Infinigence-AI & ICT, CAS & BUAA & THU & SJTU
  * Axis: **Workflow scheduling and resource reallocation**
  * Dynamically reallocates compute, memory, and communication resources across heterogeneous RL components.

* Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/qin)]
  * THU & Moonshot AI
  * Axis: **Rollout latency and long-tail mitigation**
  * Uses prompt-level context learning to predict rollout behavior and reduce long-tail latency in synchronous LLM RL.

* RobustRL: Role-Based Fault Tolerance System for RL Post-Training ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Paper](https://www.usenix.org/conference/osdi26/presentation/chen-zhenqian)]
  * ZJU & SKLMEAC
  * Axis: **Fault tolerance**
  * Provides role-based fault isolation for trainer, rollout, and management failures during RL post-training.

* Beat the long tail: Distribution-Aware Speculative Decoding for RL Training ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=kMeqqPBjSl)]
  * Together AI & Stanford & UCSD
  * Axis: **Speculative decoding for RL**; **Rollout latency and long-tail mitigation**
  * Identifies rollout length long-tail as a bottleneck and applies distribution-aware speculative decoding to accelerate generation.

* HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=LRLyuaz1W7)]
  * Amazon
  * Axis: **Heterogeneous environments**
  * Optimizes LLM reinforcement learning across heterogeneous GPU environments with multi-generation hardware.

* ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=HhDSxs7x2R)]
  * NTU & HPC-AI Tech
  * Axis: **Speculative decoding for RL**
  * Addresses drafter staleness and draft-verification misalignment when integrating speculative decoding into RL systems.

* RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching ([NSDI 2026](/reading-notes/conference/nsdi-2026)) \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/gao-wei)] \[[arXiv](https://arxiv.org/abs/2509.21009)]
  * HKUST & Alibaba
  * Axis: **Rollout latency and long-tail mitigation**
  * Packs prompts with long-tail responses into tail batches while keeping most rollout rounds balanced and short.

* Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter ([ASPLOS 2026](/reading-notes/conference/asplos-2026)) \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790231)] \[[arXiv](https://arxiv.org/abs/2511.16665)] \[[Code](https://github.com/mit-han-lab/fastrl)]
  * MIT & NVIDIA & ETH Zurich & MIT-IBM AI Lab & UMass Amherst
  * Axis: **Speculative decoding for RL**; **Rollout latency and long-tail mitigation**
  * Uses **TLT** to accelerate reasoning RL rollouts with adaptive speculative decoding, training a lightweight drafter on idle GPUs and selecting rollout strategies per batch.

* History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL ([ASPLOS 2026](/reading-notes/conference/asplos-2026)) \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790172)]
  * SJTU & ByteDance
  * Axis: **Rollout latency and long-tail mitigation**
  * Exploits reusable structure across rollout histories to reduce redundant rollout work.

* ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation ([MLSys 2025](/reading-notes/conference/mlsys-2025)) \[[Paper](https://mlsys.org/virtual/2025/poster/3228)] \[[arXiv](https://arxiv.org/abs/2406.14088)] \[[Code](https://github.com/openpsi-project/ReaLHF)]
  * THU
  * Axis: **Workflow scheduling and resource reallocation**

### Reliability and Fault Tolerance

{% hint style="info" %}
**Reading map.** These works differ along four axes:

* **Recovery mechanisms**: reduce lost work after failures through redundancy, checkpoint placement, or pre-planned reconfiguration.

* **Production reliability infrastructure**: make failures observable, diagnosable, and routinely recoverable at 10K+ GPU scale.

* **Empirical reliability studies**: characterize production failure modes and operational mitigations from large training runs.

* **Workload resilience**: absorb dynamic workload variation before it turns into large efficiency loss or training instability.
  {% endhint %}

* SDCs in the Wild: Characterizing and Diagnosing SDC-Defective GPUs in Production LLM Training ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Personal Notes](/reading-notes/conference/osdi-2026/sdchunter)] \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zheng)]
  * SJTU & ByteDance Seed
  * Axis: **Empirical reliability studies**; **Production reliability infrastructure**
  * Characterizes 23 SDC-defective GPUs from production LLM training and shows synthetic microbenchmarks miss over 60% of defective devices because SDCs can be aging-related, data-dependent, unit-specific, and invisible to ECC/thermal protections.
  * Introduces **SDCHunter**, replaying the exact training workload and input data that triggered the failure to diagnose defective GPUs.
  * Deployed at ByteDance to mitigate 40 SDC incidents, with less than 4% training overhead, 100% coverage and device-level localization for real SDC-defective accelerators, and diagnosis shortened to within one hour.

* Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours ([OSDI 2026](/reading-notes/conference/osdi-2026)) \[[Personal Notes](/reading-notes/conference/osdi-2026/aegis)] \[[Paper](https://www.usenix.org/conference/osdi26/presentation/lei)]
  * THU & ByteDance
  * Axis: **Empirical reliability studies**; **Production reliability infrastructure**
  * Presents **AEGIS**, an online SDC detection framework that decouples lightweight corruption sensing from definitive verification during large-scale LLM training.
  * Uses a two-stage `cSensor-cVerifier` design to keep runtime monitoring lightweight while confirming suspected silent corruptions.
  * Deployed over 35 million GPU hours in production, detecting 18 SDC incidents and 13 faulty GPUs with 0.86% overhead.

* MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production ([EuroSys 2026](/reading-notes/conference/eurosys-2026)) \[[Paper](https://doi.org/10.1145/3767295.3803587)] \[[arXiv](https://arxiv.org/abs/2605.08962)]
  * SJTU & ByteDance
  * Axis: **Workload resilience**
  * MLLM training can lose efficiency even without hard failures when dynamic modality and length mixtures break static encoder-backbone parallelism; decouple encoder/LLM parallelism and rebalance data loading and resharding.

* Robust LLM Training Infrastructure at ByteDance ([SOSP 2025](/reading-notes/conference/sosp-2025)) \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764838)] \[[arXiv](https://arxiv.org/abs/2509.16293)]
  * HKU & ByteDance Seed
  * Axis: **Production reliability infrastructure**
  * Large-scale LLM training needs training-aware fault management, not only generic cluster monitoring; exploit parallelism structure and training characteristics for high-capacity tolerance, fault demarcation, and localization.

* Large-Scale AI Infra Reliability: Challenges, Strategies, and Llama 3 Training Experience (DSN-S 2025) \[[Paper](https://ieeexplore.ieee.org/document/11068359)]
  * Meta
  * Axis: **Empirical reliability studies**
  * 16K-GPU training reliability is dominated by hardware failure attribution and mitigation; the contribution is a failure taxonomy and operational strategies rather than a new recovery mechanism.

* Holmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters ([NSDI 2025](/reading-notes/conference/nsdi-2025)) \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/yao)]
  * FDU & Tencent & UChicago
  * Axis: **Production reliability infrastructure**
  * Silent irregular iterations can cost more training time than fail-stop failures; Holmes localizes them in real time using abnormal communication-operator detection, communication-operator graphs, and cross-iteration analysis.

* Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures ([EuroSys 2024](/reading-notes/conference/eurosys-2024)) \[[Personal Notes](/reading-notes/conference/eurosys-2024/jit-checkpointing)] \[[Paper](https://dl.acm.org/doi/10.1145/3627703.3650085)] \[[Homepage](https://www.microsoft.com/en-us/research/publication/just-in-time-checkpointing-low-cost-error-recovery-from-deep-learning-training-failures/)]
  * MSR India
  * Axis: **Recovery mechanisms**
  * Creates recovery state only when a failure occurs, avoiding periodic checkpoint-frequency tuning and steady-state checkpoint overhead.
  * Recovers by replaying a single minibatch iteration across GPUs, reducing failure recovery from minutes to seconds per GPU with nearly zero steady-state overhead.

* MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/jiang-ziheng)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-jiang_ziheng.pdf)] \[[Code](https://github.com/volcengine/veScale)]
  * ByteDance & PKU
  * Axis: **Production reliability infrastructure**
  * Stability at 10K+ GPU scale is a full-stack property; deep observability across system components is needed to identify root causes and turn them into fault-tolerance and straggler-mitigation techniques.

* Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/abs/10.1145/3600006.3613152)] \[[arXiv](https://browse.arxiv.org/abs/2309.08125)] \[[Code](https://github.com/SymbioticLab/Oobleck)]
  * UMich SymbioticLab & AWS & PKU
  * Axis: **Recovery mechanisms**
  * Pre-generate heterogeneous pipeline templates and instantiate `f + 1` logically equivalent pipeline replicas, so failures can be handled by reconfiguration using already-replicated model states.

* Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613145)]
  * Rice & AWS
  * Axis: **Recovery mechanisms**
  * Remote storage bandwidth makes checkpoint-based recovery slow; place checkpoints in aggregate host CPU memory and schedule checkpoint traffic to preserve training throughput.

* Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/thorpe)] \[[Code](https://github.com/uclasystem/bamboo)]
  * UCLA & CMU & MSR & Princeton
  * Axis: **Recovery mechanisms**
  * Pipeline bubbles can hide redundant neighboring-layer computation, giving preemptible-instance training low-pause recovery without paying the full cost of frequent checkpointing.

## LLM Inference

### Full-Pipeline Optimization

* Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit (arXiv:2607.13095) \[[arXiv](https://arxiv.org/abs/2607.13095)]
  * MiMo Team, Xiaomi
  * Presents a production inference system for the MiMo-V2.5 family that jointly supports Hybrid Sliding Window Attention (SWA), sparse MoE, and multimodal encoders.
  * Refactors KV-cache management with separate full-attention and SWA pools, layerwise prefetch, SWA-aware prefix-cache trees, and placement policies that preserve strict $$O(W)$$ SWA storage, backed by the RDMA-optimized **GCache** distributed cache.
  * Combines KV-cache-affinity routing with three-tier **Length Bucketing** (0–64K / 64K–256K / 256K–1M) to reduce load imbalance and mixed-length prefill interference, while accelerating multimodal inputs through GPU image preprocessing, parallel video decoding, and cache sharing.

### Heterogeneous Deployment

* Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs (arXiv:2605.04357) \[[arXiv](https://arxiv.org/abs/2605.04357)]
  * CMU & PKU
  * Jointly optimizes multi-model resource allocation and per-replica serving strategies across heterogeneous cloud GPUs under throughput demand, latency SLOs, price, and resource availability.
  * Uses a lossless two-stage decomposition to preserve joint optimality while making online re-solving practical as demand and availability change.
* SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling (SIGMETRICS Abstracts 2026) \[[Paper](https://doi.org/10.1145/3771576)] \[[arXiv](https://arxiv.org/abs/2502.14617)] \[[Code](https://github.com/shashwatj07/SageServe)]
  * UIUC & GaTech & IISc & Microsoft
  * Serves mixed latency-sensitive and latency-insensitive LLM workloads across cloud regions with forecast-aware routing, GPU VM scaling, and model placement.
  * Co-optimizes short-term request routing and longer-lead-time resource allocation with traffic forecasts and ILP scheduling.
* Cauchy: A Cost-Efficient LLM Serving System through Adaptive Heterogeneous Deployment (SoCC 2025) \[[Paper](https://dl.acm.org/doi/10.1145/3772052.3772264)]
  * BUAA & Kuaishou
  * Deploys prefill and decode computation onto suitable heterogeneous GPU combos according to workload phase demands and cost efficiency.
  * Combines combo selection, hierarchical request scheduling, and dynamic autoscaling to maintain SLOs under changing request rates.
* Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (arXiv:2502.00722) \[[arXiv](https://arxiv.org/abs/2502.00722)]
  * Cambridge & HKUST & PKU & ETH & Purdue
  * Characterizes how workload mix, GPU composition, deployment configuration, and request assignment affect LLM serving cost-efficiency on heterogeneous cloud GPUs.
  * Uses mixed-integer linear programming to choose cost-efficient serving plans under price-budget and real-time GPU-availability constraints.
* HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment (ICLR 2025) \[[Paper](https://openreview.net/forum?id=Cs6MrbFuMq)] \[[arXiv](https://arxiv.org/abs/2502.07903)]
  * HKUST
  * Extends heterogeneous LLM serving to prefill-decode disaggregation, jointly placing phase computation and KV-cache communication across heterogeneous GPUs and links.
  * Combines graph partitioning and max-flow optimization to co-optimize resource allocation, per-phase parallel strategies, and inter-phase KV transfer.
* HexGen: Generative Inference of Foundation Model over Heterogeneous Decentralized Environment ([ICML 2024](/reading-notes/conference/icml-2024)) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2023/hexgen)] \[[arXiv](https://arxiv.org/abs/2311.11514)] \[[Code](https://github.com/Relaxed-System-Lab/HexGen)]
  * HKUST & ETH & CMU
  * Supports *asymmetric* tensor model parallelism and pipeline parallelism, allowing each pipeline stage to use a different layer count and tensor-parallel degree.
  * Formulates heterogeneous decentralized inference placement as a constrained optimization problem and searches layouts with a heuristic evolutionary algorithm.
* SpotServe: Serving Generative Large Language Models on Preemptible Instances ([ASPLOS 2024](/reading-notes/conference/asplos-2024)) \[[Personal Notes](/reading-notes/conference/asplos-2024/spotserve)] \[[arXiv](https://arxiv.org/abs/2311.15566)] \[[Code](https://github.com/Hsword/SpotServe)]
  * CMU & PKU & CUHK
  * Dynamically adapts LLM parallelization configurations under changing preemptible-instance availability and workload fluctuation.
  * Uses migration planning and stateful inference recovery to reduce tail latency and monetary cost on volatile GPU capacity.

### Parallelism and Partitioning

* AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://arxiv.org/abs/2302.11665)] \[[Code](https://github.com/alpa-projects/mms)]
  * UC Berkeley & PKU & UPenn & Stanford & Google
  * Trade-off between *the overhead of model parallelism* and *reduced serving latency by statistical multiplexing*.
* Efficiently Scaling Transformer Inference ([MLSys 2023](/reading-notes/conference/mlsys-2023)) \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2023/hash/523f87e9d08e6071a3bbd150e6da40fb-Abstract-mlsys2023.html)]
  * Google
  * **Outstanding Paper Award**
  * Model partitioning; PaLM; TPUv4
* DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale ([SC 2022](/reading-notes/conference/sc-2022)) \[[Paper](https://dl.acm.org/doi/abs/10.5555/3571885.3571946)] \[[Code](https://github.com/microsoft/DeepSpeed)] \[[Homepage](https://www.deepspeed.ai/inference/)]
  * Microsoft DeepSpeed
  * Leverage CPU/NVMe/GPU memory.

### LLM-based Applications

* Teola: Towards End-to-End Optimization of LLM-based Applications ([ASPLOS 2025](/reading-notes/conference/asplos-2025)) \[[arXiv](https://arxiv.org/abs/2407.00326)]
  * CUHK
  * An orchestration framework for LLM-based applications: utilize task primitives as the basic units; represent each query’s workflow as a primitive-level dataflow graph.
  * Enable larger design space for optimization including graph optimization (i.e., parallelization and pipelining) and application-aware scheduling.
* Parrot: Efficient Serving of LLM-based Applications with Semantic Variable ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/lin-chaofan)] \[[Code](https://github.com/microsoft/ParrotServe)]
  * SJTU & MSRA
* SGLang: Efficient Execution of Structured Language Model Programs ([NeurIPS 2024](/reading-notes/conference/neurips-2024)) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2024/sglang)] \[[Paper](https://openreview.net/forum?id=VqkAKQibpq)] \[[arXiv](https://arxiv.org/abs/2312.07104)] \[[Code](https://github.com/sgl-project/sglang)]
  * UC Berkeley & Stanford
  * Co-design the front-end programming interface and back-end serving runtime
  * **SGLang**; SGVM w/ **RadixAttention**
  * Reuse KV cache across multiple calls and programs

### Agentic Inference

* ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System ([ICML 2026](/reading-notes/conference/icml-2026)) \[[Paper](https://openreview.net/forum?id=kR4iOTaAOJ)] \[[arXiv](https://arxiv.org/abs/2602.13692)] \[[Code](https://github.com/ThunderAgent-org/ThunderAgent)] \[[Blog](https://thunderagent.ai)]
  * GaTech & UIUC & CMU & Together AI
  * **Spotlight**
  * Introduce **ThunderAgent**, an agentic inference and rollout system that represents workflows as LLM Programs with unified KV-cache, state, and tool-resource metadata.
  * Use program-aware scheduling and tool resource management to improve KV-cache reuse, reduce memory imbalance, and prepare tool environments asynchronously.

### Workload Characterization

* TraceLab: Characterizing Coding Agent Workloads for LLM Serving (arXiv:2606.30560) \[[arXiv](https://arxiv.org/abs/2606.30560)] \[[Code](https://github.com/uw-syfi/TraceLab)] \[[Homepage](https://tracelab.cs.washington.edu)] \[[Blog](https://syfi.cs.washington.edu/blog/2026-06-25-tracelab/)]
  * UW & Wuhan University of Technology & SJTU
  * Release a coding-agent trace with 4,265 sessions, 357,161 agent steps, and 432,510 tool calls from Claude Code and Codex usage.
  * Characterize long autonomous loops, long contexts with short outputs, heavy-tailed tool calls, and high but imperfect prefix-cache hit rates.
  * Identify serving opportunities around lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and KV-cache management around human-paced gaps.
* Measuring Agents in Production ([ICML 2026](/reading-notes/conference/icml-2026)) \[[Paper](https://openreview.net/forum?id=mWxEAgz3xu)] \[[arXiv](https://arxiv.org/abs/2512.04123)]
  * UC Berkeley & Intesa Sanpaolo & UIUC & Stanford & IBM Research
  * **Oral**
  * Present **MAP**, a systematic study of production LLM agents using 20 interviews and a survey of 86 deployed-systems practitioners across 26 domains.
  * Find that production agents often use simple, controllable approaches: bounded step counts before human intervention, off-the-shelf prompting, and human evaluation.

### Retrieval-Augmented Generation (RAG)

* CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation (arXiv:2502.11101) \[[arXiv](https://arxiv.org/abs/2502.11101)]
  * Jeonbuk National University & Seoul National University
  * Leverage query-independent, offline caching to reuse a context KV cache store.
  * *Cache Re-Positioning*: shift keys to different positions in the encoding space.
  * *Layer-Adaptive Cache Pruning*: discard low-relevance caches for documents during pre-filling.
  * *Adaptive Positional Allocation*: adjust cache positions to maximize the use of the available positional encoding range.
* Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation (SIGMOD 2025) \[[arXiv](https://arxiv.org/abs/2502.15734)]
  * Adobe Research & IIT Bombay & IIT Kanpur
  * Identify the reusability of chunk-caches; perform a small fraction of recomputation to fix the cache to maintain output quality; store and evict chunk-caches.
  * A wrapper around vLLM; built on Xformers backend optimized with Triton.
* RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation (arXiv:2404.12457) \[[arXiv](https://arxiv.org/abs/2404.12457)]
  * PKU & ByteDance
  * Organize the intermediate states of retrieved knowledge in a knowledge tree; cache them in the GPU and host memory.
  * Replacement policy: evaluate each node based on its access frequency, size, and access cost.
    * Priority= Clock + (Frequency × Cost Size) / Size
    * Nodes with lower priority are evicted first.
  * Built on vLLM.

### Request Scheduling

* Llumnix: Dynamic Scheduling for Large Language Model Serving ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/sun-biao)] \[[Code](https://github.com/AlibabaPAI/llumnix)]
  * Alibaba
* FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference ([NSDI 2026](/reading-notes/conference/nsdi-2026)) \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/wu-bingyang)] \[[arXiv](https://arxiv.org/abs/2305.05920)] \[[Code](https://github.com/LLMServe/FastServe)]
  * PKU
  * Enables iteration-level preemptive scheduling for autoregressive decoding instead of request-level FIFO execution.
  * Combines a skip-join multi-level feedback queue scheduler with proactive intermediate-state swapping to reduce head-of-line blocking.
* Orca: A Distributed Serving System for Transformer-Based Generative Models ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Personal Notes](/reading-notes/conference/osdi-2022/orca)] \[[Paper](https://www.usenix.org/conference/osdi22/presentation/yu)]
  * Seoul National University & FriendliAI
  * Iteration-level scheduling; selective batching.

### KV Cache Management

* DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants ([NSDI 2026](/reading-notes/conference/nsdi-2026)) \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/liu-yuhan)] \[[arXiv](https://arxiv.org/abs/2411.02820)]
  * UChicago & Microsoft
  * Reuses prefix KV caches across fine-tuned LLM variants with the same architecture, including across distributed nodes.
  * Selectively recomputes a small subset of layers from another model's KV cache and reuses the remaining layers with negligible quality loss.
* Jenga: Effective Memory Management for Serving LLM with Heterogeneity ([SOSP 2025](/reading-notes/conference/sosp-2025)) \[[Personal Notes](/reading-notes/conference/sosp-2025/jenga)] \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764823)] \[[arXiv](https://arxiv.org/abs/2503.18292)]
  * THU & UChicago & UC Berkeley
  * Heterogeneous embedding sizes, attention mechanisms, and token-dependency patterns in modern LLMs break fixed-page KV-cache assumptions and create fragmentation.
  * Uses a two-level memory allocator with LCM-sized compatible pages and layer-specific caching/eviction policies for heterogeneous attention patterns.
* CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion ([EuroSys 2025](/reading-notes/conference/eurosys-2025)) \[[Paper](https://doi.org/10.1145/3689031.3696098)] \[[arXiv](https://arxiv.org/abs/2405.16444)] \[[Code](https://github.com/YaoJiayi/CacheBlend)]
  * CUHK-Shenzhen & UChicago & Stanford
  * **Best Paper Award (Spring)**
  * Combines multiple precomputed KV caches for RAG inputs while selectively recomputing a small fraction of tokens to recover cross-attention.
* CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving ([SIGCOMM 2024](/reading-notes/conference/sigcomm-2024)) \[[arXiv](https://arxiv.org/abs/2310.07240)] \[[Code](https://github.com/UChi-JCL/CacheGen)] \[[Video](https://www.youtube.com/watch?v=H4_OUWvdiNo)]
  * UChicago & Microsoft & Stanford
* Prompt Cache: Modular Attention Reuse for Low-Latency Inference ([MLSys 2024](/reading-notes/conference/mlsys-2024)) \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/hash/a66caa1703fe34705a4368c3014c1966-Abstract-Conference.html)] \[[arXiv](https://arxiv.org/abs/2311.04934)]
  * Yale & Google
  * Precomputes and stores attention states for reusable prompt modules such as system messages, templates, and context documents.
  * Uses a schema to preserve positional correctness during attention-state reuse and reduce time-to-first-token without model changes.
* ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching ([ISCA 2024](/reading-notes/conference/isca-2024))
* Efficient Memory Management for Large Language Model Serving with PagedAttention ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613165)] \[[arXiv](https://browse.arxiv.org/abs/2309.06180)] \[[Code](https://github.com/vllm-project/vllm)] \[[Homepage](https://vllm.ai/)]
  * UC Berkeley & Stanford & UCSD
  * vLLM, PagedAttention
  * Partition the KV cache of each sequence into blocks, each block containing the keys and values for a fixed number of tokens

### Prefill-Decode (PD) Disaggregation

* Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter (arXiv:2604.15039) \[[arXiv](https://arxiv.org/abs/2604.15039)]
  * Moonshot AI & THU
  * Introduce **PrfaaS**, a cross-datacenter serving architecture that selectively offloads long-context prefills to standalone compute-dense clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode.
  * Combine model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement instead of fully externalizing all prefill requests.
  * Remove the requirement that heterogeneous accelerators share a single low-latency RDMA fabric; on an internal 1T-parameter hybrid model, improve serving throughput by 54% over homogeneous PD and 32% over a naive heterogeneous baseline.
* Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST 2025) \[[Paper](https://www.usenix.org/conference/fast25/presentation/qin)] \[[arXiv](https://arxiv.org/abs/2407.00079)] \[[Slides](https://www.usenix.org/system/files/fast25_slides-qin.pdf)] \[[Code](https://github.com/kvcache-ai/Mooncake)]
  * Mootshot AI & Tsinghua
  * **Best Paper Award**
  * Separate the prefill and decoding clusters; prediction-based early rejection.
  * Distributed multi-layer KVCache pool; prefix-hashed KVCache object storage.
* Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (arXiv:2401.11181) \[[arXiv](https://arxiv.org/abs/2401.11181)]
  * ICT, CAS & Huawei Cloud
* DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin)] \[[Code](https://github.com/LLMServe/DistServe)]
  * PKU & UCSD
* Splitwise: Efficient Generative LLM Inference Using Phase Splitting ([ISCA 2024](/reading-notes/conference/isca-2024)) \[[Paper](https://ieeexplore.ieee.org/document/10609649/)] \[[arXiv](https://arxiv.org/abs/2311.18677)] \[[Blog](https://www.microsoft.com/en-us/research/blog/splitwise-improves-gpu-usage-by-splitting-llm-inference-phases/)]
  * UW & Microsoft
  * **Best Paper Award**
  * Split the two phases (i.e., prefill and decode) of a LLM inference request to separate machines

### Chunked Prefill

* LMPrefill: An Inference Engine for Prefill-only Workloads in Large Language Model Applications ([SOSP 2025](/reading-notes/conference/sosp-2025)) \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764834)] \[[arXiv](https://arxiv.org/abs/2505.07203)]
  * UChicago & THU & LinkedIn & UC Berkeley
  * Targets LLM applications that generate only one output token and avoids storing KV caches for all layers.
  * Combines hybrid prefilling, suffix KV cache discarding/offloading, and JCT-aware scheduling for prefill-only requests.
* Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/agrawal)] \[[Code](https://github.com/microsoft/sarathi-serve)] \[[arXiv](https://arxiv.org/abs/2403.02310)]
  * MSR India & GaTech
  * **Sarathi-Serve**

### Serverless Inference

* HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds ([NSDI 2026](/reading-notes/conference/nsdi-2026)) \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/lou)] \[[arXiv](https://arxiv.org/abs/2502.15524)] \[[Code](https://github.com/LLMServe/hydraserve)]
  * PKU & Alibaba Cloud
  * Minimizes serverless LLM cold-start latency in public clouds through proactive model distribution, overlapped worker startup, and GPU-network-contention-aware worker placement.
  * Consolidates pipelines to reduce cold-start resource usage while improving SLO attainment.
* FaaScale: Unlocking Fast LLM Scaling for Serverless Inference ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=jgL8LuOVyT)] \[[arXiv](https://arxiv.org/abs/2502.09922)]
  * CUHK-SZ & UVA & HKUST & Alibaba & Nokia Bell Labs
  * Formerly **λScale**; enables fast model scaling for serverless LLM inference with pipelined multicast inference.
  * Uses **PipeCast** to adaptively multicast model blocks and dynamically form cross-node inference pipelines during model transfer.
* ServerlessLLM: Low-Latency Serverless Inference for Large Language Models ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/fu)] \[[Code](https://github.com/ServerlessLLM/ServerlessLLM)] \[[arXiv](https://arxiv.org/abs/2401.14351)]
  * Edinburgh

### LoRA Serving

* dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/wu-bingyang)]
  * PKU & Shanghai AI Lab
* CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference (arXiv:2401.11240) \[[arXiv](https://arxiv.org/abs/2401.11240)]
  * HKUST & CUHK-Shenzhen & Shanghai AI Lab & Huawei Cloud
* S-LoRA: Serving Thousands of Concurrent LoRA Adapters ([MLSys 2024](/reading-notes/conference/mlsys-2024)) \[[arXiv](https://arxiv.org/abs/2311.03285)] \[[Code](https://github.com/S-LoRA/S-LoRA)]
  * UC Berkeley
* Punica: Multi-Tenant LoRA Serving ([MLSys 2024](/reading-notes/conference/mlsys-2024)) \[[arXiv](https://arxiv.org/abs/2310.18547)] \[[Code](https://github.com/punica-ai/punica)]
  * UW & Duke

### Position-Independent Caching (PIC)

* EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models (ICML 2025) \[[arXiv](https://arxiv.org/abs/2410.15332)]
  * PKU & NJU & Huawei Cloud
  * Key insight: the initial tokens of each chunk separately absorb a disproportionate amount of attention, preventing subsequent tokens from attending to relevant parts.
  * Propose an algorithm named *LegoLink* to recompute k (≤ 32) initial tokens on each chunk (except the first chunk) → Recognize their non-initial status and cripple their attention-absorbing ability.
  * Compared to CacheBlend, LegoLink reduces recomputation complexity and relies on static attention sparsity.

### Compression

* Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization ([ISCA 2024](/reading-notes/conference/isca-2024))

### Sparsity

* InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/lee)]
  * Seoul National University
* PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU ([SOSP 2024](/reading-notes/conference/sosp-2024)) \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695964)] \[[arXiv](https://arxiv.org/abs/2312.12456)] \[[Code](https://github.com/SJTU-IPADS/PowerInfer)]
  * SJTU
  * A GPU-CPU hybrid inference engine
  * Hot-activated neurons are preloaded onto the GPU for fast access; cold-activated neurons are computed on the CPU
* Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time ([ICML 2023](/reading-notes/conference/icml-2023)) \[[Paper](https://proceedings.mlr.press/v202/liu23am.html)] \[[Code](https://github.com/FMInference/DejaVu)]
  * Rice & ZJU & Stanford & UCSD & ETH & Adobe & Meta AI & CMU
  * A system to predict *contextual sparsity* (small, input-dependent sets that yield *approximately* the same output).

### Speculative Decoding

* Online Speculative Decoding ([ICML 2024](/reading-notes/conference/icml-2024)) \[[arXiv](https://arxiv.org/abs/2310.07177)]
  * UC Berkeley & UCSD & Sisu Data & SJTU
* SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification ([ASPLOS 2024](/reading-notes/conference/asplos-2024)) \[[arXiv](https://arxiv.org/abs/2305.09781)] \[[Code](https://github.com/flexflow/FlexFlow/tree/inference)]
  * CMU
* Speculative Decoding with Big Little Decoder (NeurIPS 2023) \[[Paper](https://arxiv.org/abs/2302.07863)]
  * UC Berkeley & ICSI & LBNL
* Fast Inference from Transformers via Speculative Decoding ([ICML 2023](/reading-notes/conference/icml-2023)) \[[Paper](https://openreview.net/pdf?id=C9NEblP8vS)]
  * Google Research

### Offloading

* LLM in a flash: Efficient Large Language Model Inference with Limited Memory (arXiv 2312.11514) \[[arXiv](https://arxiv.org/abs/2312.11514)]
  * Apple
* FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU ([ICML 2023](/reading-notes/conference/icml-2023)) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2023/flexgen)] \[[Paper](https://proceedings.mlr.press/v202/sheng23a.html)] \[[Code](https://github.com/FMInference/FlexGen)]
  * Stanford & UC Berkeley & ETH & Yandex & HSE & Meta & CMU
  * *High-throughput serving; only use a single GPU.*

### Fairness

* Locality-aware Fair Scheduling in LLM Serving (arXiv:2501.14312) \[[arXiv](https://arxiv.org/abs/2501.14312)]
  * UC Berkeley
* Fairness in Serving Large Language Models ([OSDI 2024](/reading-notes/conference/osdi-2024)) \[[Paper](https://www.usenix.org/conference/osdi24/presentation/sheng)] \[[Code](https://github.com/Ying1123/VTC-artifact)]
  * UC Berkeley

## LLM Alignment

* PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch ([ATC 2024](/reading-notes/conference/atc-2024)) \[[Paper](https://www.usenix.org/conference/atc24/presentation/lei)]
  * THU

## Acronyms

* LLM: Large Language Model
* LoRA: Low-Rank Adaptation
* MoE: Mixture-of-Experts
* RL: Reinforcement Learning
* RLHF: Reinforcement Learning from Human Feedback
* SDC: Silent Data Corruption


# Diffusion Models

## Diffusion Model Serving

### Image Generation

* DiFlow: A System for Micro-Serving Text-to-Image Diffusion Workflows ([SOSP 2026](/reading-notes/conference/sosp-2026)) \[[arXiv](https://arxiv.org/abs/2604.08123)]
  * HKUST & Alibaba
  * **Our work!**
  * Decomposes text-to-image diffusion workflows into independently managed model-execution nodes.
  * Enables per-model scaling, cross-workflow model sharing, and adaptive model parallelism for bursty serving workloads.
* MixFusion: A Patch-Level Parallel Serving System for Mixed-Resolution Diffusion Models ([PPoPP 2026](/reading-notes/conference/ppopp-2026)) \[[Paper](https://doi.org/10.1145/3774934.3786420)] \[[arXiv](https://arxiv.org/abs/2501.09253)] \[[Code](https://github.com/desenSunUBW/mixfusion)]
  * UWaterloo & CMU & Rice
  * Uses patch-level parallelism to batch mixed-resolution text-to-image diffusion requests.
  * Combines patch-tailored cache management with SLO-aware scheduling and online latency prediction.
* xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism (arXiv:2411.01738) \[[arXiv](https://arxiv.org/abs/2411.01738)] \[[Code](https://github.com/xdit-project/xDiT)]
  * Tencent
  * Several parallel approaches for DiTs.
* Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters ([ATC 2025](/reading-notes/conference/atc-2025)) \[[Paper](https://www.usenix.org/conference/atc25/presentation/li-suyi-katz)] \[[arXiv](https://arxiv.org/abs/2407.02031)] \[[Code](https://github.com/modelscope/Katz)] \[[Trace](https://modelscope.cn/datasets/mental2008/T2I-Model-Serving-Request-Trace)]
  * HKUST & Alibaba
  * **Our work!**
  * ControlNet-as-a-Service decouples ControlNets from the base model for caching, parallelization, and sharing.
  * Bounded Asynchronous Loading overlaps LoRA loading with initial denoising steps; latent parallelism accelerates the base model across GPUs.
* PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Diffusion Transformer Models (arXiv:2405.14430) \[[arXiv](https://arxiv.org/abs/2405.14430)] \[[Code](https://github.com/xdit-project/xDiT)]
  * Tencent & HKU
* Cache Me if You Can: Accelerating Diffusion Models through Block Caching ([CVPR 2024](/reading-notes/conference/cvpr-2024)) \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Wimbauer_Cache_Me_if_You_Can_Accelerating_Diffusion_Models_through_Block_CVPR_2024_paper.html)] \[[Homepage](https://fwmb.github.io/blockcaching/)]
  * Meta & TUM & MCML & Oxford
* DeepCache: Accelerating Diffusion Models for Free ([CVPR 2024](/reading-notes/conference/cvpr-2024)) \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Ma_DeepCache_Accelerating_Diffusion_Models_for_Free_CVPR_2024_paper.html)] \[[Code](https://github.com/horseee/DeepCache)]
  * NUS
* DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models ([CVPR 2024](/reading-notes/conference/cvpr-2024)) \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Li_DistriFusion_Distributed_Parallel_Inference_for_High-Resolution_Diffusion_Models_CVPR_2024_paper.html)] \[[Code](https://github.com/mit-han-lab/distrifuser)]
  * MIT & Princeton & Lepton AI & NVIDIA
  * Split the model input into multiple patches and assign each patch to a GPU.
* Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/agarwal-shubham)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-agarwal_shubham.pdf)]
  * Adobe Research & UIUC
  * Skip a certain number of denoising steps.

### Video Generation

* Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization ([ICML 2026](/reading-notes/conference/icml-2026)) \[[arXiv](https://arxiv.org/abs/2602.02958)] \[[Code](https://github.com/svg-project/Quant-VideoGen)]
  * UC Berkeley & MIT & NVIDIA & Amazon & UT Austin
  * Introduces **QVG**, a training-free 2-bit KV-cache quantization framework for auto-regressive video diffusion models.
  * Combines Semantic-Aware Smoothing with Progressive Residual Quantization to reduce KV-cache memory by up to 7.0x with less than 4% end-to-end latency overhead.
* StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=p9WALNBvc6)] \[[arXiv](https://arxiv.org/abs/2511.07399)] \[[Code](https://github.com/chenfengxu714/StreamDiffusionV2)] \[[Homepage](https://streamdiffusionv2.github.io/)]
  * UT Austin & UC Berkeley & Nunchaku AI & Stanford & First Intelligence & MIT & Shizuku AI
  * Introduces a training-free streaming pipeline that brings video diffusion models to interactive live generation under time-to-first-frame and per-frame latency SLOs.
  * Combines SLO-aware batching and block scheduling, a rolling KV cache, motion-aware noise control, and multi-GPU pipeline orchestration.
* Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (NeurIPS 2025) \[[arXiv](https://arxiv.org/abs/2505.18875)] \[[Code](https://github.com/svg-project/Sparse-VideoGen)]
  * UC Berkeley & MIT & NVIDIA & Stanford
  * Introduces **SVG2**, a training-free sparse-attention framework that clusters and reorders video tokens by semantic similarity.
  * Uses semantic-aware permutation, dynamic top-p budget control, and customized kernels to improve the quality-efficiency trade-off without padding scattered critical tokens.
* Fast Video Generation with Sliding Tile Attention ([ICML 2025](/reading-notes/conference/icml-2025)) \[[Paper](https://proceedings.mlr.press/v267/zhang25m.html)] \[[OpenReview](https://openreview.net/forum?id=U74MOXPEJd)] \[[arXiv](https://arxiv.org/abs/2502.04507)] \[[Code](https://github.com/hao-ai-lab/FastVideo)]
  * UCSD & UMich & THU & UC Berkeley & MBZUAI
  * Introduces **STA**, a tile-wise local spatiotemporal attention mechanism for video DiTs that exploits localized attention score structure.
  * Uses a hardware-aware sliding-window design and kernel optimizations to speed up video diffusion attention and end-to-end HunyuanVideo generation.
* Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity ([ICML 2025](/reading-notes/conference/icml-2025)) \[[Paper](https://proceedings.mlr.press/v267/xi25c.html)] \[[arXiv](https://arxiv.org/abs/2502.01776)] \[[Code](https://github.com/svg-project/Sparse-VideoGen)]
  * UC Berkeley & MIT & NVIDIA & THU
  * Introduces **SVG**, a training-free framework that dynamically classifies video DiT attention heads by spatial or temporal sparse patterns.
  * Combines online attention profiling with hardware-aware tensor layouts and customized kernels to accelerate video generation while preserving quality.
* Looking Backward: Streaming Video-to-Video Translation with Feature Banks (ICLR 2025) \[[arXiv](https://arxiv.org/abs/2405.15757)] \[[Code](https://github.com/Jeff-LiangF/streamv2v)] \[[Homepage](https://jeff-liangf.github.io/projects/streamv2v/)]
  * UT Austin & UC Berkeley
  * Introduces **StreamV2V**, a real-time streaming video-to-video translation system that supports unbounded input streams without diffusion-model fine-tuning.
  * Maintains a compact feature bank of past frames and extends self-attention and feature fusion to preserve temporal consistency at 20 FPS on one A100 GPU.
* FlexCache: Flexible Approximate Cache System for Video Diffusion (arXiv:2501.04012) \[[arXiv](https://arxiv.org/abs/2501.04012)]
  * UWaterloo
  * Cache for text-to-video diffusion models.

### Image Editing

* FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling ([EuroSys 2026](/reading-notes/conference/eurosys-2026)) \[[Paper](https://doi.org/10.1145/3767295.3769379)] \[[arXiv](https://arxiv.org/abs/2505.20600)] \[[Code](https://github.com/Sylvia-16/FlashPS)]
  * HKUST & Alibaba
  * **Our work!**
  * Uses mask-aware caching and scheduling to accelerate generative image editing workloads.
* CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model ([CVPR 2024](/reading-notes/conference/cvpr-2024)) \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Zeng_CAT-DM_Controllable_Accelerated_Virtual_Try-on_with_Diffusion_Model_CVPR_2024_paper.html)] \[[Code](https://github.com/zengjianhao/CAT-DM)]
  * TJU & Tencent
  * **CAT-DM**: **C**ontrollable **A**ccelerated virtual **T**ry-on with **D**iffusion **M**odel

## Diffusion Model Training

* DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines ([MLSys 2024](/reading-notes/conference/mlsys-2024)) \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/45c1f6a8cbf2da59ebf2c802b4f742cd-Paper-Conference.pdf)] \[[Slides](https://mlsys.org/media/mlsys-2024/Slides/2636_X1xLHC2.pdf)]
  * HKU & AWS & OSU
  * Fill the computation of non-trainable model parts into idle periods of the pipeline training of the backbones.

## Supporting Add-on Modules

* X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model ([CVPR 2024](/reading-notes/conference/cvpr-2024)) \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Ran_X-Adapter_Adding_Universal_Compatibility_of_Plugins_for_Upgraded_Diffusion_Model_CVPR_2024_paper.html)] \[[Homepage](https://showlab.github.io/X-Adapter/)] \[[Code](https://github.com/showlab/X-Adapter)]
  * NUS & Tencent & FDU

## Domain-Specific Accelerator (DSA)

* Cambricon-D: Full-Network Differential Acceleration for Diffusion Models ([ISCA 2024](/reading-notes/conference/isca-2024)) \[[Paper](https://ieeexplore.ieee.org/document/10609724)]
  * ICT, CAS

## Acronyms

* DiT: Diffusion Transformer


# Deep Learning Recommendation Model (DLRM)

## DLRM Training

* Heterogeneous Acceleration Pipeline for Recommendation System Training ([ISCA 2024](/reading-notes/conference/isca-2024)) \[[arXiv](https://arxiv.org/abs/2204.05436)]
  * UBC & GaTech
  * **Hotline**: a runtime framework.
  * Utilize CPU main memory for non-popular embeddings and GPUs’ HBM for popular embeddings.
  * Fragment a mini-batch into popular and non-popular micro-batches (μ-batches).
* Accelerating Neural Recommendation Training with Embedding Scheduling ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/zeng)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-zeng.pdf)] \[[Code](https://github.com/HKUST-SING/herald)]
  * HKUST
  * **Herald**: an adaptive location-aware inputs allocator to determine *where embeddings should be trained* and an optimal communication plan generator to determine *which embeddings should be synchronized*.
* Bagpipe: Accelerating Deep Recommendation Model Training ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/abs/10.1145/3600006.3613142)]
  * UW-Madison & UChicago

## DLRM Inference

* DisaggRec: Architecting Disaggregated Systems for Large-Scale Personalized Recommendation (arXiv 2212.00939) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2022/disaggrec)] \[[Paper](https://arxiv.org/abs/2212.00939)]
  * Meta AI & WashU & UPenn & Cornell & Intel
  * *Disaggregated* system; *decouple* CPUs and memory resources; *partition embedding tables*.

## Pruning

* AdaEmbed: Adaptive Embedding for Large-Scale Recommendation Models ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/lai)]
  * UMich SymbioticLab & Meta
  * In-training pruning.

## GPU Cache

* UGache: A Unified GPU Cache for Embedding-based Deep Learning ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Personal Notes](/reading-notes/conference/sosp-2023/ugache)] \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613169)]
  * SJTU
  * A *unified multi-GPU cache* system.
  * Used for GNN training and DLR inference.
* EVStore: Storage and Caching Capabilities for Scaling Embedding Tables in Deep Recommendation Systems ([ASPLOS 2023](/reading-notes/conference/asplos-2023)) \[[Personal Notes](/reading-notes/conference/asplos-2023/evstore)] \[[Paper](https://dl.acm.org/doi/10.1145/3575693.3575718)] \[[Code](https://github.com/ucare-uchicago/ev-store-dlrm)]
  * UChicago & Beijing University of Technology & Bandung Institute of Technology, Indonesia & Seagate Technology & Emory
  * A *caching* layer optimized for embedding *access patterns*.

## Model Update

* Ekko: A Large-Scale Deep Learning Recommender System with Low-Latency Model Update ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Paper](https://www.usenix.org/conference/osdi22/presentation/sima)]
  * Tencent & Edinburgh
  * P2P model update dissemination.

## Acronyms

* DLRM: Deep Learning Recommendation Model


# Mixture of Experts (MoE)

## MoE Training

* Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models ([SIGCOMM 2023](/reading-notes/conference/sigcomm-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3603269.3604869)]
  * THU & ByteDance
* Accelerating Distributed MoE Training and Inference with Lina ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/li-jiamin)]
  * CityU & ByteDance & CUHK
* SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/zhai)] \[[Code](https://github.com/thu-pacman/SmartMoE-AE)]
  * THU

## MoE Inference

* Accelerating Distributed MoE Training and Inference with Lina ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/li-jiamin)]
  * CityU & ByteDance & CUHK
* Optimizing Dynamic Neural Networks with Brainstorm ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/cui)]
  * SJTU & MSRA & USTC

## Models

* Mixtral-8x7B \[[Hugging Face](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)] \[[Blog](https://mistral.ai/news/mixtral-of-experts/)]
  * Mistral AI


# Hyper-Parameter Tuning (HPO)

{% hint style="info" %}
No active maintenance.
{% endhint %}

## Optimizing HPO Workloads

* Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/hu)] \[[Code](https://github.com/S-Lab-System-Group/Hydro)]
  * NTU & Shanghai AI Lab
  * Surrogate models; co-locate HPO jobs with LLM training.
* Elastic Hyperparameter Tuning on the Cloud ([SoCC 2021](/reading-notes/conference/socc-2021)) \[[Paper](https://dl.acm.org/doi/10.1145/3472883.3486989)]
  * UC Berkeley
  * SEER
* RubberBand: Cloud-based Hyperparameter Tuning ([EuroSys 2021](/reading-notes/conference/eurosys-2021)) \[[Paper](https://dl.acm.org/doi/10.1145/3447786.3456245)]
  * UC Berkeley

## HPO for Systems

* Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving ([SoCC 2021](/reading-notes/conference/socc-2021)) \[[Paper](https://dl.acm.org/doi/10.1145/3472883.3486987)] \[[Code](https://github.com/kubedl-io/morphling)]
  * HKUST & Alibaba
  * Meta-learning for tuning resources and runtime parameters in model serving systems.
* Selecting the Best VM across Multiple Public Clouds: A Data-Driven Performance Modeling Approach ([SoCC 2017](/reading-notes/conference/socc-2017)) \[[Paper](https://doi.org/10.1145/3127479.3131614)]
  * UC Berkeley & FAIR & MSR
  * Select the best VM across multiple cloud providers.
* CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics ([NSDI 2017](/reading-notes/conference/nsdi-2017)) \[[Paper](https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/alipourfard)]
  * Yale & MSR & UC Berkeley & Alibaba
  * Pick the right cloud configuration for *big data analytics jobs*.
* Google Vizier: A Service for Black-Box Optimization (KDD 2017) \[[Paper](https://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization)]
  * Google
  * Black-box optimization.

## Acronyms

* HPO: Hyper-Parameter Tuning


# Reinforcement Learning (RL)

* MSRL: Distributed Reinforcement Learning with Dataflow Fragments ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/zhu-huanzhou)] \[[Code](https://github.com/mindspore-lab/mindrl)]
  * ICL & Huawei
  * Fragmented dataflow graph.
* Ray: A Distributed Framework for Emerging AI Applications ([OSDI 2018](/reading-notes/conference/osdi-2018)) \[[Paper](https://www.usenix.org/conference/osdi18/presentation/moritz)] \[[Code](https://github.com/ray-project/ray)] \[[Homepage](https://www.ray.io/)]
  * UC Berkeley
  * RL framework


# Deep Learning Compiler

{% hint style="info" %}
No active maintenance.
{% endhint %}

## System Architecture

* MLIR: Scaling Compiler Infrastructure for Domain Specific Computation (CGO 2021) \[[Paper](https://research.google/pubs/pub49988/)] \[[Homepage](https://mlir.llvm.org/)]
  * Google
* TVM: An Automated End-to-End Optimizing Compiler for Deep Learning ([OSDI 2018](/reading-notes/conference/osdi-2018)) \[[Paper](https://www.usenix.org/conference/osdi18/presentation/chen)] \[[Code](https://github.com/apache/tvm)] \[[Homepage](https://www.usenix.org/conference/osdi18/presentation/chen)]
  * UW & AWS & SJTU & UC Davis & Cornell

## Tensor Program Generation

### Megakernel Compilation

* Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://mlsys.org/virtual/2026/poster/3592)] \[[arXiv](https://arxiv.org/abs/2604.13327)]
  * CMU
  * Introduce **Event Tensor**, a unified compiler abstraction that represents tiled-task dependencies as first-class tensors for dynamic megakernels.
  * Support both symbolic-shape dynamism and data-dependent dynamism, then lower the abstraction through the **Event Tensor Compiler (ETC)** with static and dynamic scheduling transformations.
  * Achieve state-of-the-art LLM serving latency, including up to 1.40x speedup on fused GEMM and Reduce-Scatter kernels, up to 1.23x on MoE workloads, and up to 3.5x lower engine warmup overhead.
* Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs (arXiv:2512.22219) \[[arXiv](https://arxiv.org/abs/2512.22219)] \[[Code](https://github.com/mirage-project/mirage)] \[[Homepage](https://catalyst.cs.cmu.edu/projects/mpk.html)]
  * CMU
  * Introduce **Mirage Persistent Kernel (MPK)**, the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance megakernel.
  * Use an SM-level graph representation and an in-kernel parallel runtime with decentralized scheduling to enable cross-operator software pipelining and fine-grained kernel overlap.
  * Reduce end-to-end LLM inference latency by up to 1.7x over kernel-per-operator serving systems while preserving flexible tensor-programming workflows.

### General Tensor Program Generation

* DITRON: Distributed Multi-level Tiling Compiler for Parallel Tensor Programs ([ICML 2026](/reading-notes/conference/icml-2026)) \[[arXiv](https://arxiv.org/abs/2605.02953)] \[[Code](https://github.com/ByteDance-Seed/Triton-distributed)]
  * ByteDance Seed & PKU & THU & ZJU & SJTU
  * Introduce **DITRON**, a distributed tensor compiler with Core-, Device-, and Task-level tiling interfaces that map computation and communication onto the memory hierarchy of GPU clusters.
  * Reorder tiles through distributed swizzling to overlap computation with communication, and use task dependencies and scheduling to fuse registered Triton kernels into megakernels.
  * Support NVIDIA and AMD GPUs through hardware-specific SHMEM backends; report 6-30% speedups on isolated kernels and 5-30% gains in end-to-end vLLM inference over expert-tuned CUDA baselines.
* Cocktailer: Analyzing and Optimizing Dynamic Control Flow in Deep Learning ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/zhang-chen)]
  * THU & MSRA
  * Co-optimize the execution of control flow and data flow.
* Welder: Scheduling Deep Learning Memory Access via Tile-graph ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/shi)]
  * PKU & MSRA
  * Optimize memory access.
* Effectively Scheduling Computational Graphs of Deep Neural Networks toward Their Domain-Specific Accelerators ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/zhao)]
  * Stream Computing
  * GraphTurbo: Scheduler for DSA.
* EINNET: Optimizing Tensor Programs with Derivation-Based Transformations ([OSDI 2023](/reading-notes/conference/osdi-2023)) \[[Paper](https://www.usenix.org/conference/osdi23/presentation/zheng)]
  * THU & CMU
  * Leverage *transformations* between general tensor algebra expressions.
* AStitch: Enabling a New Multi-dimensional Optimization Space for Memory-Intensive ML Training and Inference on Modern SIMT Architectures (ASPLOS 2022) \[[Paper](https://dl.acm.org/doi/10.1145/3503222.3507723)]
  * Alibaba
  * Memory-intensive operators.
* Ansor: Generating High-Performance Tensor Programs for Deep Learning ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Paper](https://www.usenix.org/conference/osdi20/presentation/zheng)]
  * UC Berkeley

## Acronyms

* DSA: Domain-Specific Architecture


# Deep Learning Framework

* Pathways: Asynchronous Distributed Dataflow for ML (MLSys 2022) \[[Paper](https://mlsys.org/virtual/2022/oral/2146)]
  * Google
  * **Outstanding Paper Award**
* OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (arXiv 2110.15032) \[[Paper](https://arxiv.org/abs/2110.15032)] \[[Code](https://github.com/Oneflow-Inc/oneflow)] \[[中文官网](https://www.oneflow.org/index.html)]
  * OneFlow
* Jittor: a novel deep learning framework with meta-operators and unified graph execution (Science China Information Sciences 2020) \[[Paper](http://scis.scichina.com/en/2020/222103.pdf)] \[[Code](https://github.com/Jittor/Jittor)] \[[中文主页](https://cg.cs.tsinghua.edu.cn/jittor/)]
  * THU
* PyTorch: An Imperative Style, High-Performance Deep Learning Library (NeurIPS 2019) \[[Paper](https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html)] \[[Code](https://github.com/pytorch/pytorch)] \[[Homepage](https://pytorch.org/)]
  * FAIR
* XDL: An Industrial Deep Learning Framework for High-dimensional Sparse Data (DLP-KDD 2019) \[[Paper](https://dl.acm.org/doi/10.1145/3326937.3341255)] \[[Code](https://github.com/alibaba/x-deeplearning)]
  * Alibaba
  * High-dimensional sparse data.
* TensorFlow: A System for Large-Scale Machine Learning (OSDI 2016) \[[Paper](https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi)] \[[Code](https://github.com/tensorflow/tensorflow)] \[[Homepage](https://www.tensorflow.org/)]
  * Google Brain
* MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems (NIPS 2016 Workshop on MLSys) \[[Paper](https://arxiv.org/abs/1512.01274)] \[[Homepage](https://mxnet.apache.org/)] \[[Code](https://github.com/apache/mxnet)]
* Caffe: Convolutional Architecture for Fast Feature Embedding (arXiv 1408.5093) \[[Paper](https://arxiv.org/abs/1408.5093)] \[[Homepage](http://caffe.berkeleyvision.org/)] \[[Code](https://github.com/BVLC/Caffe/)]
  * UC Berkeley
* An Introduction to Computational Networks and the Computational Network Toolkit (MSR-TR-2014-112) \[[Paper](https://www.microsoft.com/en-us/research/publication/an-introduction-to-computational-networks-and-the-computational-network-toolkit/)] \[[Code](https://github.com/microsoft/CNTK)] \[[Homepage](https://learn.microsoft.com/en-us/cognitive-toolkit/)]
  * Microsoft
  * CNTK
  * No longer developed.


# Cloud-Edge Collaboration

{% hint style="warning" %}
No active maintenance.
{% endhint %}

## Framework

* Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Paper](https://www.usenix.org/conference/osdi22/presentation/lv)] \[[Code](https://github.com/alibaba/MNN)] \[[中文官网](http://www.mnn.zone/)]
  * Alibaba
  * Production system; support for collaborative ML; optimized for mobile devices.

## Automatic Graph Partitioning

* SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud (MobiCom 2020) \[[Paper](https://dl.acm.org/doi/10.1145/3372224.3419194)]
  * Samsung AI Center & Cambridge
  * Early-exit; split CNNs at runtime.
* Dynamic Adaptive DNN Surgery for Inference Acceleration on the Edge (INFOCOM 2019) \[[Paper](https://ieeexplore.ieee.org/document/8737614)]
  * PolyU & Sydney
  * DNN surgery: Consider different network conditions; characterize DNNs as DAG rather than a chain.
* Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices (ICDCS 2017) \[[Paper](https://ieeexplore.ieee.org/document/7979979)] \[[Code](https://github.com/kunglab/ddnn)]
  * Harvard
  * DDNN: Distribute DNN across the cloud, the edge and end devices.
* Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge ([ASPLOS 2017](/reading-notes/conference/asplos-2017)) \[[Personal Notes](/reading-notes/conference/asplos-2017/neurosurgeon)] \[[Paper](https://dl.acm.org/doi/10.1145/3037697.3037698)]
  * UMich
  * At the granularity of NN layers; select *one* partition point.


# ML for Systems

* [Agentic System Generation](/paper-list/ml-for-systems/agentic-system-generation)
* [Kernel Generation](/paper-list/ml-for-systems/kernel-generation)


# Agentic System Generation

Papers on using LLMs or coding agents to synthesize and optimize end-to-end systems and application deployments.

## Serving System Generation

* FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications (arXiv:2607.18171) \[[arXiv](https://arxiv.org/abs/2607.18171)] \[[Code](https://github.com/Infini-AI-Lab/FlashRT)] \[[Homepage](https://infini-ai-lab.github.io/flashrt-blog)]
  * CMU & AMD & University at Buffalo
  * Introduces **FlashRT**, an agent harness that transforms simple single-GPU multimodal reference applications into optimized multi-GPU deployments.
  * Uses a chain-of-program workflow to derive and validate an intermediate representation, identify candidate transformations through static analysis, and iteratively implement, verify, and benchmark them.
  * Optimizes placement, streaming, and intra-model parallelism across hardware budgets and latency or throughput objectives.
* VibeServe: Can AI Agents Build Bespoke LLM Serving Systems? (arXiv:2605.06068) \[[arXiv](https://arxiv.org/abs/2605.06068)] \[[Code](https://github.com/uw-syfi/vibe-serve)]
  * UW
  * Introduces **VibeServe**, a multi-agent loop that synthesizes complete LLM serving stacks for a target model, workload, and hardware configuration.
  * Uses an outer loop to plan and track the system-design search and an inner loop to implement candidates, check correctness, and benchmark performance.
  * Remains competitive with vLLM in standard deployments and exploits specialization opportunities in six non-standard serving scenarios.


# Kernel Generation

Papers on using LLMs or agents for kernel generation, kernel verification, tensor program generation, and compiler optimization.

## Agent-Based Kernel Generation

* Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent (arXiv:2607.14541) \[[arXiv](https://arxiv.org/abs/2607.14541)] \[[Code](https://github.com/alibaba/atrex-kernel-agent)] \[[Benchmark](https://github.com/alibaba/atrex-bench)]
  * Alibaba
  * Present **Atrex-Kernel-Agent (AKA)**, a GPU kernel optimization agent that combines iterative measurement and revision with profiler feedback and a layered knowledge base of reference kernels and optimization techniques.
  * Use optimization dropout to mask stale iteration memories while preserving the accepted kernel and audit trail, allowing a fresh agent context to explore alternative optimization directions.
  * In controlled case studies, replace PyTorch fallbacks with generated FlyDSL kernels that match or exceed hand-tuned production baselines.
* KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta (ISCA 2026) \[[Personal Notes](/reading-notes/conference/isca-2026/kernelevolve)] \[[arXiv](https://arxiv.org/abs/2512.23236)] \[[Blog](https://engineering.fb.com/2026/04/02/developer-tools/kernelevolve-how-metas-ranking-engineer-agent-optimizes-ai-infrastructure/)]
  * Meta
  * Present **KernelEvolve**, an agentic kernel coding framework that automates kernel generation and optimization from kernel specifications for recommendation workloads across heterogeneous accelerators.
  * Search over kernels across multiple programming abstractions, from Triton and CuTe DSL to low-level hardware-agnostic languages, using graph-based search with runtime-aware retrieval-augmented prompt synthesis.
  * Validate 100% pass rates on all 250 KernelBench problems and on 160 PyTorch ATen operators across NVIDIA GPUs, AMD GPUs, and Meta accelerators, while reducing kernel development time from weeks to hours.
* AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=SBS4NJHYjZ)]
  * Stanford & UofT & AWS
  * Present **AccelOpt**, a self-improving LLM agentic system that optimizes AI accelerator kernels through iterative generation and optimization memory.
* Agentic Operator Generation for ML ASICs ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=O3Bx0nNGnW)]
  * Meta
  * Present **TritorX**, an agentic AI system that generates correct Triton kernels for Meta's MTIA accelerator at scale.
  * Use LLM-driven generation with compilation-based validation to target an ML ASIC backend rather than only commodity GPUs.
* Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems ([MLSys 2026](/reading-notes/conference/mlsys-2026)) \[[Paper](https://openreview.net/forum?id=MJxhiX3sSd)]
  * LBNL
  * Study LLM-based multi-agent systems for GPU kernel optimization and compare their dynamics against existing compilers and manual development.
* ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants (arXiv:2604.18616) \[[arXiv](https://arxiv.org/abs/2604.18616)]
  * CausalFlow Inc. & HKUST & THU & Stanford & UCAS & UC Riverside
  * Present **ARGUS**, an agentic GPU kernel optimization framework that uses compile-time data-flow invariants to turn global kernel constraints into dense, structured feedback.
  * Provide a tile-based Pythonic DSL with propagated tags and tag assertions, verifying invariants through abstract interpretation and SMT solving while returning concrete counterexamples for targeted fixes.
  * On AMD MI300X GPUs, generated GEMM, flash-attention, and MoE kernels reach 99-104% of state-of-the-art hand-optimized assembly throughput; ARGUS also solves 100% of KernelBench Level 1 and 90% of Level 2 tasks.
* Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance (arXiv:2603.29010) \[[arXiv](https://arxiv.org/abs/2603.29010)]
  * NVIDIA
  * Introduce two efficiency principles for LLM-based GPU kernel optimization agents: a compact domain-specific language and Speed-of-Light (SOL) guidance.
  * Implement these ideas in **muCUTLASS**, a CUTLASS-backed DSL that exposes kernel configuration, epilogue fusion, and multi-stage pipeline choices while SOL analysis steers search, budgets trials, and flags benchmark gaming.
  * On 59 KernelBench problems, moving from low-level code generation to DSL code raises GPT-5-mini from a 0.40x regression to a 1.27x speedup over PyTorch; adding SOL-guided steering increases this to 1.56x while SOL-guided budgeting saves 19-43% of tokens.
* AVO: Agentic Variation Operators for Autonomous Evolutionary Search (arXiv:2603.24517) \[[arXiv](https://arxiv.org/abs/2603.24517)]
  * NVIDIA
  * Introduce **AVO**, a family of evolutionary variation operators that replaces fixed mutation, crossover, and hand-designed heuristics with autonomous coding agents.
  * Treat variation as a self-directed agent loop that consults the current lineage, a domain-specific knowledge base, and execution feedback to propose, repair, critique, and verify kernel edits.
  * Run 7 days of autonomous evolution on attention kernels for NVIDIA Blackwell B200 GPUs, outperforming cuDNN by up to 3.5% and FlashAttention-4 by up to 10.5%, with additional transfer gains on grouped-query attention after 30 minutes of adaptation.
* AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Search (arXiv:2603.21331) \[[arXiv](https://arxiv.org/abs/2603.21331)] \[[Code](https://github.com/RightNow-AI/autokernel)]
  * RightNow AI
  * Introduce an autonomous GPU kernel optimizer that starts from end-to-end PyTorch models and iteratively optimizes the most impactful kernels.
  * Use Amdahl's law to rank kernel opportunities and a five-stage correctness pipeline to validate candidate optimizations.
  * Report strong kernel-level and end-to-end model speedups across matmul, attention, convolution, and MLP workloads on H100 GPUs.
* Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis (arXiv:2603.10846) \[[arXiv](https://arxiv.org/abs/2603.10846)] \[[Homepage](https://evokernel.zhuo.li/)] \[[Dataset](https://huggingface.co/datasets/noahli/EvoKernel)]
  * SJTU & Shanghai AI Lab & MemTensor
  * Introduce **EvoKernel**, a self-evolving agentic framework for NPU kernel synthesis in data-scarce programming domains.
  * Formulate synthesis as a memory-based reinforcement learning task with value-driven retrieval and cross-task memory sharing for cold-start drafting and continual latency refinement.
  * Improve frontier models' correctness from 11.0% to 83.0% and achieve a median speedup of 3.60x over initial drafts on an NPU variant of KernelBench.
* CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation (arXiv:2602.24286) \[[arXiv](https://arxiv.org/abs/2602.24286)] \[[Code](https://github.com/BytedTsinghua-SIA/CUDA-Agent)] \[[Homepage](https://cuda-agent.github.io/)]
  * ByteDance Seed & Tsinghua AIR
  * Present a large-scale agentic RL system for high-performance CUDA kernel generation.
  * Combine scalable data synthesis, a skill-augmented CUDA development environment, and stable RL training for verifiable kernel optimization.
  * Outperform `torch.compile` by 100%, 100%, and 92% on the three KernelBench levels.

## Kernel Verification

* ProofWright: Towards Agentic Formal Verification of CUDA (arXiv:2511.12294) \[[arXiv](https://arxiv.org/abs/2511.12294)] \[[Homepage](https://mast.stanford.edu/pubs/proofwright_towards_agentic_formal_verification_of_cuda/)]
  * Georgia Tech & NVIDIA & Stanford
  * Introduce **ProofWright**, an agentic formal-verification framework that establishes memory safety, thread safety, and semantic correctness for LLM-generated CUDA kernels.
  * Combine a VerCors agent that synthesizes safety annotations with a Rocq-based framework that translates PyTorch specifications and proves semantic equivalence.
  * Verify memory safety and data-race freedom for 74% of KernelBench L1 kernels and semantic equivalence for 14%, with an average overhead of about three minutes per kernel.

## Compiler Optimization

* Meta Large Language Model Compiler: Foundation Models of Compiler Optimization \[[Paper](https://ai.meta.com/research/publications/meta-large-language-model-compiler-foundation-models-of-compiler-optimization/)]
  * Meta AI

## Benchmarks

* CommBench: Can LLMs Write Correct and Efficient GPU Communication Code? (arXiv:2608.04450) \[[arXiv](https://arxiv.org/abs/2608.04450)] \[[Code](https://github.com/uccl-project/CommBench)] \[[Blog](https://uccl-project.github.io/posts/commbench/)]
  * UC Davis & UC Berkeley & UIUC & CUHK & UCSD & UW-Madison
  * Present **CommBench**, a benchmark of over 100 expert-curated GPU communication tasks covering point-to-point transfers, collectives, expert parallelism, compute-communication fusion, and utilities.
  * Automate compilation, execution, and validation on multi-GPU systems with checks against benchmark cheating, jointly measuring correctness and performance across intra-node NVLink and inter-node RDMA.
  * In the paper's evaluation, GPT-5.5 produces correct code with competitive performance on only 30.7% of tasks, exposing gaps between LLM-generated and expert-written communication code.
* Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent (arXiv:2607.14541) \[[arXiv](https://arxiv.org/abs/2607.14541)] \[[Benchmark](https://github.com/alibaba/atrex-bench)]
  * Alibaba
  * Introduce **Atrex-Bench**, with 30 operators and 440 shapes sampled from production inference traces of GPUs with limited compute capacity and abundant memory.
  * Score kernels against per-problem roofline ceilings, weighting results by each operator's share of GPU time per serving phase and each application's share of fleet card-hours to reflect production importance.
  * Evaluate six coding agents: the best unaugmented model reaches only about 10% of the hardware roofline, while PyTorch fallbacks inflate apparent correctness rates.
* SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits (arXiv:2603.19173) \[[arXiv](https://arxiv.org/abs/2603.19173)] \[[Code](https://github.com/NVIDIA/SOL-ExecBench)] \[[Benchmark](https://research.nvidia.com/benchmarks/sol-execbench)]
  * NVIDIA
  * Present a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models, targeting NVIDIA Blackwell GPUs.
  * Measure candidate kernels against analytically derived Speed-of-Light bounds via **SOLAR**, using a SOL Score that quantifies how much of the remaining gap to hardware-efficient execution is closed.
  * Provide a sandboxed evaluation harness with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static-analysis-based checks against reward hacking.
* TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (arXiv:2502.14752) \[[arXiv](https://arxiv.org/abs/2502.14752)] \[[Benchmark](https://github.com/thunlp/TritonBench)]
  * THU-NLP
  * Present the first comprehensive benchmark for Triton operator generation.
  * Cover two evaluation channels: 184 real-world operators from GitHub and a collection of operators aligned with PyTorch interfaces.
  * Evaluate both functional correctness and efficiency on widely deployed GPUs.
* KernelBench: Can LLMs Write Efficient GPU Kernels? (arXiv:2502.10517) \[[arXiv](https://arxiv.org/abs/2502.10517)] \[[Benchmark](https://github.com/ScalingIntelligence/KernelBench)] \[[Homepage](https://scalingintelligence.stanford.edu/blogs/kernelbench/)]
  * Stanford
  * Provide an open-source benchmark for evaluating whether LMs can generate fast and correct GPU kernels for PyTorch ML workloads.
  * Cover 250 tasks spanning single-kernel operators, simple fusion patterns, and full model architectures.
  * Introduce `fast_p`, a metric that measures the fraction of generated kernels that are both correct and faster than a configurable baseline threshold.


# Artificial Intelligence (AI)

## Training Methods

* [Continual Pre-training](/paper-list/artificial-intelligence/cpt)
* [On-Policy Distillation](/paper-list/artificial-intelligence/opd)

## Models

* [Diffusion Models](/paper-list/artificial-intelligence/diffusion-models)
* [Language Models](/paper-list/artificial-intelligence/language-models)
* [Deep Learning Recommendation Models](/paper-list/artificial-intelligence/dlrm)


# Continual Pre-training

> **Continual Pre-training (CPT)** updates a pretrained language model on new corpora before downstream adaptation. The usual goal is to add domain, language, temporal, or agentic capability while controlling forgetting and retraining cost.

## Core Methods

* ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning (arXiv:2510.10071) \[[arXiv](https://arxiv.org/abs/2510.10071)] \[[Code](https://github.com/PuppyKnightUniversity/ADEPT)]
  * PKU & ZJU
  * Expands only layers that are less critical to general-domain competence, then applies unit-wise asymmetric learning rates.
  * Targets domain-adaptive CPT with less parameter update and less interference than full-parameter CPT.
* Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models (CoLLAs 2025) \[[arXiv](https://arxiv.org/abs/2508.01908)]
  * UdeM & Mila & IBM Research & Fujitsu Research & Polytechnique Montreal
  * Studies replay and gradient alignment for Llama-family CPT across language streams with 100B tokens per language.
  * Shows low replay rates and efficient meta-experience replay can stabilize learning without the overhead of large replay budgets.
* Beyond Cosine Decay: On the Effectiveness of Infinite Learning Rate Schedule for Continual Pre-training (CoLLAs 2025) \[[arXiv](https://arxiv.org/abs/2503.02844)] \[[Code](https://github.com/Pauljanson002/beyond-cosine)]
  * Mila & Concordia & UdeM
  * Compares repeated cosine annealing with infinite learning-rate schedules for continual self-supervised pretraining.
  * Removes the need to precommit to a fixed token budget and reduces forgetting from LR re-warming.
* Efficient Continual Pre-training by Mitigating the Stability Gap (arXiv:2406.14833) \[[arXiv](https://arxiv.org/abs/2406.14833)]
  * PKU & HKUST & MIT-IBM Watson AI Lab
  * Identifies the early "stability gap" where domain performance drops before recovering during CPT.
  * Mitigates the gap through multi-epoch subset training, high-quality subset selection, and data mixtures closer to the original pretraining distribution.
* Simple and Scalable Strategies to Continually Pre-train Large Language Models (TMLR 2024) \[[Paper](https://openreview.net/forum?id=DimPeeCxKO)] \[[arXiv](https://arxiv.org/abs/2403.08763)]
  * UdeM & Concordia & Mila & EleutherAI
  * Combines LR re-warming, LR re-decaying, and replay of previous data for English-to-English and English-to-German CPT.
  * Shows the recipe can approach retraining-from-scratch baselines at 405M scale and in a 10B-model setting.
* Continual Pre-training of Language Models (ICLR 2023) \[[arXiv](https://arxiv.org/abs/2302.03241)] \[[Code](https://github.com/UIC-Liu-Lab/ContinualLM)]
  * UIC & PKU & KDDI Research
  * Formalizes continual domain-adaptive pretraining over a sequence of unlabeled domain corpora.
  * Introduces DAS with soft masking and contrastive knowledge integration to reduce forgetting while preserving transfer.

## Data Selection and Mixture

* Midtraining Bridges Pretraining and Posttraining Distributions (arXiv:2510.14865) \[[arXiv](https://arxiv.org/abs/2510.14865)]
  * CMU
  * Frames midtraining as distributional bridging: mixing specialized data with general pretraining data before posttraining.
  * Shows math/code midtraining can outperform specialized continued pretraining while reducing forgetting; timing and mixture weight interact through a plasticity window.
* Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training (arXiv:2507.15640) \[[arXiv](https://arxiv.org/abs/2507.15640)]
  * Manchester & MSR & ICL & UCLA
  * Trains a small agent to learn domain re-weighting heuristics from data-mixing trajectories and evaluation feedback.
  * Generalizes data-mixture control across source fields, target models, and domain spaces without retraining the agent.

## Scaling Laws and Training Dynamics

* Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025) \[[Paper](https://proceedings.mlr.press/v267/wang25cx.html)] \[[arXiv](https://arxiv.org/abs/2505.07796)]
  * UCAS & IA, CAS & RitzzAI
  * Models CPT loss curves as a transition between original and domain-specific training trajectories.
  * Decouples distribution shift and LR annealing to predict loss over CPT steps, replay ratios, and schedules.
* D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models (NeurIPS 2024) \[[Paper](https://openreview.net/forum?id=JzKFN5fWOk)] \[[arXiv](https://arxiv.org/abs/2406.01375)]
  * Alibaba Group & University of Waterloo & Manchester & QMUL & HKUST & M-A-P
  * Predicts general and downstream performance as a function of domain/general mixture ratio, model size, and dataset size.
  * Extends the law to cross-domain settings, reducing the search cost for target-domain mixture ratios.

## Domain, Language, and Agent Adaptation

* Model Spec Midtraining: Improving How Alignment Training Generalizes (arXiv:2605.02087) \[[arXiv](https://arxiv.org/abs/2605.02087)]
  * Anthropic Fellows Program & Anthropic
  * Introduces MSM: train on synthetic documents discussing the Model Spec after pretraining and before alignment fine-tuning.
  * Shapes how models generalize from the same demonstration data; reduces Qwen3-32B agentic misalignment from 54% to 7% in the reported setup.
* Scaling Agents via Continual Pre-training (arXiv:2509.13310) \[[arXiv](https://arxiv.org/abs/2509.13310)] \[[Code](https://github.com/Alibaba-NLP/DeepResearch)] \[[Homepage](https://tongyi-agent.github.io/blog)]
  * Tongyi Lab, Alibaba Group
  * Adds Agentic CPT before post-training so the base model already contains tool-use and deep-research behaviors.
  * Builds AgentFounder with first-order and higher-order action synthesis plus subsequent agentic alignment.
* Disentangling Continued Pre-Training: Attention-Driven Routing and Semantic Hub Preservation in Language Adaptation (Findings ACL 2026) \[[Paper](https://aclanthology.org/2026.findings-acl.1218/)] \[[Code](https://github.com/ReML-AI/disentangling_cpt)]
  * University College Cork
  * Analyzes continued pretraining for language adaptation through attention routing and semantic hub preservation.
  * Provides a mechanism-level view of how language adaptation changes internal routing and shared representations.
* Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models Through Continual Pre-Training (arXiv:2502.06589) \[[arXiv](https://arxiv.org/abs/2502.06589)]
  * Georgia Tech & Amazon
  * Introduces Hephaestus-Forge, a 103B-token agent pretraining corpus covering APIs, function-calling trajectories, reasoning, and feedback adaptation.
  * Uses CPT to improve fundamental agent capabilities before task-specific prompting or finetuning.
* Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation (ACL 2025) \[[Paper](https://aclanthology.org/2025.acl-long.1547/)]
  * HiTZ Center, UPV/EHU & Reka AI
  * Shows English data in the mixture may be critical for downstream emergent abilities even when validation perplexity looks unchanged.
  * Uses language-agnostic ICL evaluation to expose early forgetting during target-language CPT.
* Towards Effective and Efficient Continual Pre-training of Large Language Models (ACL 2025) \[[Paper](https://aclanthology.org/2025.acl-long.289/)] \[[arXiv](https://arxiv.org/abs/2407.18743)] \[[Code](https://github.com/RUC-GSAI/Llama-3-SynE)]
  * RUC & UCSD
  * Presents Llama-3-SynE, a transparent CPT recipe for improving Chinese ability and scientific reasoning on Llama-3-8B.
  * Uses data curation, synthetic scientific QA, curriculum design, and performance-tracking mixture adjustment.
* Efficient Continual Pre-training for Building Domain Specific Large Language Models (Findings ACL 2024) \[[Paper](https://aclanthology.org/2024.findings-acl.606/)]
  * Amazon
  * Builds FinPythia through domain-adaptive CPT for finance.
  * Tests task-aware and task-agnostic data selection strategies that use a small fraction of the corpus while preserving open-domain ability.

## Analysis and Benchmarks

* How Do Large Language Models Learn Concepts During Continual Pre-Training? (arXiv:2601.03570) \[[arXiv](https://arxiv.org/abs/2601.03570)]
  * UC Davis & Virginia Tech & UCLA & Meta AI
  * Links concept learning and forgetting to internal concept circuits and graph metrics.
  * Studies interference and synergy across concepts during two-stage CPT.
* Continual Pre-training of MoEs: How Robust Is Your Router? (TMLR 2025) \[[Paper](https://openreview.net/forum?id=dR7C1K71Rs)] \[[arXiv](https://arxiv.org/abs/2503.05029)]
  * UdeM & Mila & Concordia & UChicago & Capital One
  * Studies dense and MoE transformers under CPT to test whether routing worsens forgetting or load imbalance.
  * Finds MoE routers remain robust under distribution shift and can retain sample-efficiency benefits during CPT.
* Investigating Continual Pretraining in Large Language Models: Insights and Implications (TMLR 2025) \[[Paper](https://openreview.net/forum?id=aKjJoEVKgO)] \[[arXiv](https://arxiv.org/abs/2402.17400)]
  * University of Tubingen & Cohere for AI
  * Studies continual domain-adaptive pretraining across domain sequences and model scales.
  * Finds domain semantic similarity, training order, and model size affect specialization, forward transfer, backward transfer, and forgetting.

## Acronyms

* ADEPT: Adaptive Expansion and Dynamic Decoupled Tuning
* AFT: Alignment Fine-Tuning
* CPT: Continual Pre-training
* DAP: Domain-Adaptive Pre-training
* DAS: Continual DA-pre-training of LMs with Soft-masking
* D-CPT: Domain-specific Continual Pre-training
* EMA: Exponential Moving Average
* ICL: In-Context Learning
* LR: Learning Rate
* MER: Meta-Experience Replay
* MoE: Mixture-of-Experts
* MSM: Model Spec Midtraining


# On-Policy Distillation

> **On-Policy Distillation (OPD)**: student model generates its own rollouts; teacher provides per-token / per-sequence supervision (typically reverse KL) on those self-generated trajectories. Contrasts with off-policy SeqKD (SFT on teacher-generated text). Conceptually equivalent to "RL with the teacher's log-prob as a dense reward."

## Core Methods

* MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training (arXiv:2606.30406) \[[arXiv](https://arxiv.org/abs/2606.30406)]
  * PKU & Xiaomi LLM Core & HKU & RUC
  * Integrates multiple domain RL teachers by distilling each teacher on student-generated rollouts with dense token-level supervision.
  * Outperforms Mix-RL, Cascade RL, off-policy finetuning, and parameter merging on Qwen3-30B-A3B; reports deployment in MiMo-V2-Flash.
* DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025) \[[arXiv](https://arxiv.org/abs/2503.07067)]
  * KAIST & MSR
  * Couples loss formulation to data source (different loss on teacher- vs. student-generated tokens); contrastive objective compounds OPD gains.
* BOND: Aligning LLMs with Best-of-N Distillation (arXiv:2407.14622) \[[arXiv](https://arxiv.org/abs/2407.14622)]
  * Google DeepMind
  * OPD variant where the **teacher is a reward model's implicit Best-of-N policy**, distilled into the student via Jeffreys-divergence on student rollouts.
* DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024) \[[arXiv](https://arxiv.org/abs/2402.03898)] \[[Code](https://github.com/jongwooko/distillm)]
  * KAIST & MSR
  * **Skew-KL** (bounded-gradient variant of reverse-KL) + adaptive off-policy mix to cut per-step rollout cost.
* On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes / GKD (ICLR 2024) \[[arXiv](https://arxiv.org/abs/2306.13649)]
  * Google DeepMind
  * **Generalized Knowledge Distillation (GKD)**: unifies SFT, SeqKD, and OPD as a mixture of teacher- and student-generated data with arbitrary divergence (JSD, fwd/rev KL).
  * Shows on-policy student samples are essential at high compression ratios. *The canonical OPD reference.*
* MiniLLM: Knowledge Distillation of Large Language Models (ICLR 2024) \[[arXiv](https://arxiv.org/abs/2306.08543)] \[[Code](https://github.com/microsoft/LMOps/tree/main/minillm)]
  * THU & MSR
  * Replace forward-KL (mode-covering) with **reverse-KL** optimized via policy-gradient on student rollouts.
  * Mode-seeking reduces hallucination and exposure bias; the first widely-cited OPD recipe for modern LLMs.

## Analysis

* Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes (arXiv:2603.25562) \[[arXiv](https://arxiv.org/abs/2603.25562)] \[[Code](https://github.com/hhh675597/revisiting_opd)] \[[Blog](https://yuqianfu.notion.site/revisiting-opd)]
  * CASIA & UCAS & Fudan
  * Analyzes why sampled-token OPD is brittle: imbalanced token-level signal, unreliable teacher guidance on student-generated prefixes, and tokenizer / special-token mismatch.
  * Proposes teacher top-K local support matching with truncated reverse-KL, top-p rollout sampling, and special-token masking for more stable OPD.
* Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting (arXiv:2510.18874) \[[arXiv](https://arxiv.org/abs/2510.18874)]
  * Princeton
  * On-policy data is the active ingredient in RL's forgetting resistance; on-policy SFT alone closes much of the gap to full RL.
* RL's Razor: Why Online Reinforcement Learning Forgets Less (arXiv:2509.04259) \[[arXiv](https://arxiv.org/abs/2509.04259)]
  * MIT
  * On-policy training (incl. OPD) is biased toward KL-minimal solutions; explains why it preserves prior skills vs. SFT.

## Production Recipes

* MiMo-V2-Flash Technical Report (arXiv:2601.02780) \[[arXiv](https://arxiv.org/abs/2601.02780)] \[[Code](https://github.com/XiaomiMiMo)]
  * Xiaomi LLM-Core
  * **Multi-Teacher On-Policy Distillation (MOPD)**: multiple domain-specialized RL teachers provide dense token-level reward; student perfectly inherits expertise.
  * 309B total / 15B active MoE; matches DeepSeek-V3.2 and Kimi-K2 with 1/2 and 1/3 of total parameters.
  * MTP repurposed as speculative-decoding draft: 3.6 acceptance length, 2.6× decoding speedup.
* On-Policy Distillation (Thinking Machines Lab Blog, 2025-10) \[[Blog](https://thinkingmachines.ai/blog/on-policy-distillation/)] \[[Code](https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/distillation)]
  * Thinking Machines Lab
  * Treat per-token reverse KL as a dense reward; one-line change on top of an RL trainer (swap KL regularizer for teacher).
  * Replicates Qwen3: **AIME'24 60 → 70 % in \~150 steps, ≈9–30× cheaper than RL**.
  * Framing: `OPD = on-policy + dense`; `SFT = off-policy + dense`; `RL = on-policy + sparse`.
  * Re-energized the systems/ML community around OPD in late 2025.
* Qwen3 Technical Report (arXiv:2505.09388) \[[arXiv](https://arxiv.org/abs/2505.09388)]
  * Alibaba Qwen Team
  * Reports OPD reaching **74.4 % AIME'24 vs. 67.6 % for RL at 1/10 the GPU hours** — the result that inspired the TML blog.
  * First public production demonstration that OPD beats RL on the small reasoning model line.

## Acronyms

* GKD: Generalized Knowledge Distillation
* KD: Knowledge Distillation
* KL: Kullback–Leibler (divergence)
* MoE: Mixture-of-Experts
* MOPD: Multi-Teacher On-Policy Distillation
* MTP: Multi-Token Prediction
* OPD: On-Policy Distillation
* RL: Reinforcement Learning
* SeqKD: Sequence-Level Knowledge Distillation
* SFT: Supervised Fine-Tuning


# Diffusion Models

## Image Generation

### Diffusion Transformer (DiT)

* FLUX.1 \[[Code](https://github.com/black-forest-labs/flux)]
  * Black Forest Labs
  * Text-to-image generation
  * Models
    * FLUX.1-dev: <https://huggingface.co/black-forest-labs/FLUX.1-dev>
    * FLUX.1-schnell: <https://huggingface.co/black-forest-labs/FLUX.1-schnell>
* Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv:2403.03206) \[[arXiv](https://arxiv.org/abs/2403.03206)] \[[Blog](https://stability.ai/news/stable-diffusion-3)]
  * Stability AI
  * **Stable Diffusion 3 (SD3)**
  * Multimodal Diffusion Transformer (MMDiT)
  * Models
    * Stable Diffusion 3 Medium: <https://huggingface.co/stabilityai/stable-diffusion-3-medium>
* Scalable Diffusion Models with Transformers (ICCV 2023) \[[arXiv](https://arxiv.org/abs/2212.09748)] \[[Paper](https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_Diffusion_Models_with_Transformers_ICCV_2023_paper.html)] \[[Code](https://github.com/facebookresearch/DiT)] \[[Homepage](https://www.wpeebles.com/DiT)]
  * UC Berkeley & NYU
  * **DiT**

### UNet

* Kolors: Effective Training of Diffusion Model for Photorealistic Text-to-Image Synthesis \[[Technical Report](https://github.com/Kwai-Kolors/Kolors/blob/master/imgs/Kolors_paper.pdf)]
  * Kuaishou Kolors
  * Text-to-image generation
  * Model: <https://huggingface.co/Kwai-Kolors/Kolors>
* SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (arXiv:2307.01952) \[[arXiv](https://arxiv.org/abs/2307.01952)]
  * Stability AI
  * Models
    * <https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0>
    * <https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0>
* High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022) \[[Paper](https://openaccess.thecvf.com/content/CVPR2022/html/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper)] \[[arXiv](https://arxiv.org/abs/2112.10752)] \[[Code](https://github.com/CompVis/stable-diffusion)]
  * LMU Munich & Runway ML
  * Latent Diffusion Models (LDMs)
  * Models
    * Stable-Diffusion-v1-5: <https://huggingface.co/runwayml/stable-diffusion-v1-5>
      * Initialized with the weights of the **Stable-Diffusion-v1-2** checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512.

## Video Generation

* Stable Video 4D (SV4D)
  * Stability AI
  * Model: <https://huggingface.co/stabilityai/sv4d>
    * Generate **40** frames (5 video frames x 8 camera views) at 576x576 resolution, given 5 reference frames of the same size.
* Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (arXiv:2311.15127) \[[arXiv](https://arxiv.org/abs/2311.15127)] \[[Blog](https://stability.ai/news/stable-video-diffusion-open-ai-video-model)]
  * Stability AI
  * **Stable Video Diffusion** (SVD)
  * Text-to-video and image-to-video generation
  * Models
    * <https://huggingface.co/stabilityai/stable-video-diffusion-img2vid>
      * Generate **14** frames at resolution **576x1024** given a context frame of the same size.
    * <https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt>
      * Fine-tuned from the SVD-img2vid.
      * Generate **25** frames at resolution **576x1024** given a context frame of the same size.

## Acronyms

* LLM: Large Language Model


# Language Models

* Grok-2 \[[Blog](https://x.ai/blog/grok-2)]
  * xAI
  * Grok-2 Beta was released on 2024/08/13.
* Gemma 2: Improving Open Language Models at a Practical Size (arXiv:2408.00118) \[[arXiv](https://arxiv.org/abs/2408.00118)] \[[Code](https://github.com/google-deepmind/gemma)]
  * Gemma Team, Google DeepMind
  * **Gemma 2**
  * Models: <https://www.kaggle.com/models/google/gemma>
* The Llama 3 Herd of Models (arXiv:2407.21783) \[[arXiv](https://arxiv.org/abs/2407.21783)] \[[Blog](https://ai.meta.com/blog/meta-llama-3/)] \[[Code](https://github.com/meta-llama/llama3)]
  * MetaAI
  * **Llama 3**
  * Models
    * Llama 3 8B: <https://huggingface.co/meta-llama/Meta-Llama-3-8B>
    * Llama 3 70B
    * Llama 3 405B
* Mixtral 8x7B (arXiv:2401.04088) \[[arXiv](https://arxiv.org/abs/2401.04088)] \[[Blog](https://mistral.ai/news/mixtral-of-experts/)] \[[Code](https://github.com/mistralai/mistral-inference)]
  * Mistral AI
  * **Mixtral 8x7B**
  * Model: <https://huggingface.co/mistralai/Mixtral-8x7B-v0.1>
* Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv 2307.09288) \[[Paper](https://arxiv.org/abs/2307.09288)] \[[Homepage](https://ai.meta.com/llama/)]
  * Meta AI
  * **Llama 2**
  * Released with a *permissive* community license and is available for commercial use.
* LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) \[[Paper](https://arxiv.org/abs/2302.13971)] \[[Code](https://github.com/facebookresearch/llama)]
  * Meta AI
  * **6.7B, 13B, 32.5B, 65.2B**
  * Open-access
* PaLM: Scaling Language Modeling with Pathways (JMLR 2023) \[[Paper](https://www.jmlr.org/papers/v24/22-1144.html)] \[[PaLM API](https://developers.googleblog.com/2023/03/announcing-palm-api-and-makersuite.html)]
  * **540B**; open access to PaLM APIs in March 2023.
* BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (arXiv 2211.05100) \[[Paper](https://arxiv.org/abs/2211.05100)] \[[Model](https://huggingface.co/bigscience/bloom)] \[[Blog](https://bigscience.huggingface.co/blog/bloom)]
  * **176B**
  * open-access
* OPT: Open Pre-trained Transformer Language Models (arXiv: 2205.01068) \[[Paper](https://arxiv.org/abs/2205.01068)] \[[Code](https://github.com/facebookresearch/metaseq/tree/main/projects/OPT)]
  * Meta AI
  * Range from 125M to 175B parameters.
  * Open-access


# Deep Learning Recommendation Model (DLRM)

* Efficient Long Sequential User Data Modeling for Click-Through Rate Prediction (DLP-KDD 2022) \[[Paper](https://arxiv.org/abs/2209.12212)]
  * Alibaba
  * ETA: *Efficient target attention* network
  * Locality-sensitive hashing
  * Deployed on Taoba&#x6F;*.*
* Wide & Deep Learning for Recommender Systems (DLRS 2016) \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2016/wide-and-deep-learning-for-recommender-systems)] \[[Paper](https://dl.acm.org/doi/10.1145/2988450.2988454)]
  * Google
  * WDL: Wide & Deep model


# Hardware Virtualization

* [GPU Sharing](/paper-list/hardware-virtualization/gpu-sharing)


# GPU Sharing

{% hint style="info" %}
I am actively maintaining this list.
{% endhint %}

## GPU Temporal Sharing

* Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications ([EuroSys 2024](/reading-notes/conference/eurosys-2024)) \[[Personal Notes](/reading-notes/conference/eurosys-2024/orion-interference-aware-fine-grained-gpu-sharing-for-ml-applications)] \[[Paper](https://anakli.inf.ethz.ch/papers/orion_eurosys24.pdf)]
  * ETH
  * Intercept GPU kernel launches and schedule individual GPU operators
  * Utilize CUDA stream priorities; consider the PCIe bandwidth
  * Use NVIDIA Nsight Compute and NVIDIA Nsight Systems to collect the compute throughput, memory throughput, and execution time of each kernel
* Interference-aware Multiplexing for Deep Learning in GPU Clusters: A Middleware Approach ([SC 2023](/reading-notes/conference/sc-2023)) \[[Personal Notes](/reading-notes/conference/sc-2023/iadeep)] \[[Paper](https://doi.org/10.1145/3581784.3607060)] \[[Code](https://github.com/buzy-coder/IADeep)]
  * UMacau & SIAT, CAS
  * IADeep — a cluster scheduler on top of Kubernetes
  * Tune training configurations (e.g., batch size) across all co-located tasks; choose appropriate tasks to multiplex on a GPU device; consider PCIe bandwidth
* Transparent GPU Sharing in Container Clouds for Deep Learning Workloads ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/wu)] \[[Code](https://github.com/pkusys/TGS)]
  * PKU
  * TGS: *Transparent* GPU sharing; adaptive rate control; unified memory.
* Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences ([OSDI 2022](/reading-notes/conference/osdi-2022)) \[[Personal Notes](/reading-notes/conference/osdi-2022/reef)] \[[Paper](https://www.usenix.org/conference/osdi22/presentation/han)] \[[Code](https://github.com/SJTU-IPADS/reef)] \[[Benchmark](https://github.com/SJTU-IPADS/disb)] \[[Artifact](https://github.com/SJTU-IPADS/reef-artifacts/tree/osdi22-ae)]
  * SJTU
  * REEF: GPU kernel preemption; dynamic kernel padding.
* Gemini: Enabling Multi-Tenant GPU Sharing Based on Kernel Burst Estimation ([TCC 2021](/reading-notes/journal/tcc/tcc-2021)) \[[Paper](https://ieeexplore.ieee.org/document/9566822)] \[[Code](https://github.com/NTHU-LSALAB/Gemini)]
  * National Tsing Hua University
  * Enable fine-grained GPU allocation; launch kernels together.
* AntMan: Dynamic Scaling on GPU Clusters for Deep Learning ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Paper](https://www.usenix.org/conference/osdi20/presentation/xiao)] \[[Code](https://github.com/alibaba/GPU-scheduler-for-deep-learning)]
  * Alibaba
  * Enable GPU sharing in DL frameworks (TensorFlow/PyTorch); schedule operators.
* KubeShare: A Framework to Manage GPUs as First-Class and Shared Resources in Container Cloud ([HPDC 2020](/reading-notes/conference/hpdc-2020)) \[[Personal Notes](/reading-notes/conference/hpdc-2020/kubeshare)] \[[Paper](https://dl.acm.org/doi/10.1145/3369583.3392679)] \[[Code](https://github.com/NTHU-LSALAB/KubeShare)]
  * National Tsing Hua University
  * Kubernetes; CUDA API remoting.
* GaiaGPU: Sharing GPUs in Container Clouds ([ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018](/reading-notes/conference/ispa-iucc-bdcloud-socialcom-sustaincom-2018)) \[[Personal Notes](/reading-notes/conference/ispa-iucc-bdcloud-socialcom-sustaincom-2018/gaiagpu)] \[[Paper](https://ieeexplore.ieee.org/document/8672318)] \[[Code](https://github.com/tkestack/gpu-manager)]
  * PKU & Tencent
  * Kubernetes; CUDA API remoting.

## GPU Spatial Sharing

* Paella: Low-latency Model Serving with Software-defined GPU Scheduling ([SOSP 2023](/reading-notes/conference/sosp-2023)) \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613163)] \[[Code](https://github.com/eniac/paella)]
  * UPenn & DBOS, Inc.
  * Instrument kernels to expose at runtime, detailed information about the occupancy and utilization of the GPU's SMs
  * Compiler-library-scheduler co-design
* MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters (arXiv 2303.13803) \[[Paper](https://arxiv.org/abs/2303.13803)]
  * PKU & ByteDance
  * Utilize NVIDIA MPS
* NVIDIA Multi-Instance GPU (MIG) \[[Homepage](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/)]
  * Partition the GPU into as many as *seven instances*, each *fully isolated* with its own high-bandwidth memory, cache, and compute cores.
  * Available for NVIDIA H100, A100, and A30 GPUs.
* NVIDIA Multi-Process Service (MPS) \[[Docs](https://docs.nvidia.com/deploy/mps/index.html)]
  * Transparently enable co-operative multi-process CUDA applications.
  * Terminating an MPS client without synchronizing with all outstanding GPU work (via Ctrl-C / program exception such as segfault / signals, etc.) can leave the MPS server and other MPS clients in an undefined state, which may result in hangs, unexpected failures, or corruptions.
* NVIDIA CUDA Multi-Stream \[[Docs](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html)]
  * Stream: a sequence of operations that execute in issue-order on the GPU.
  * Perform multiple CUDA operations *simultaneously*.

## Survey

* GPU Virtualization and Scheduling Methods: A Comprehensive Survey ([CSUR 2017](/reading-notes/journal/csur/csur-2017)) \[[Personal Notes](/reading-notes/journal/csur/csur-2017/gpu-virtualization-survey)] \[[Paper](https://dl.acm.org/doi/10.1145/3068281)]
  * Queen’s University Belfast


# Resource Disaggregation

* [GPU Disaggregation](/paper-list/resource-disaggregation/gpu-disaggregation)
* [Memory Disaggregation](/paper-list/resource-disaggregation/memory-disaggregation)


# GPU Disaggregation

## RPC

* Slashing the Disaggregation Tax in Heterogeneous Data Centers with FractOS ([EuroSys 2022](/reading-notes/conference/eurosys-2022)) \[[Personal Notes](/reading-notes/conference/eurosys-2022/slashing-the-disaggregation-tax-in-heterogeneous-data-centers-with-fractos)] \[[Paper](https://dl.acm.org/doi/10.1145/3492321.3519569)]
  * ICL & Technion & Barkhausen Institut & TU Dresden
  * A *GPU adaptor* offers several RPCs: GPU context initialization, memory de/allocation, kernel loading, kernel invocation, and cleanup.

## Remote CUDA

* DGSF: Disaggregated GPUs for Serverless Functions ([IPDPS 2022](/reading-notes/conference/ipdps-2022)) \[[Personal Notes](/reading-notes/conference/ipdps-2022/dgsf)] \[[Paper](https://ieeexplore.ieee.org/document/9820659)]
  * UT-Austin & Katana Graph
  * Serverless functions; intercept CUDA APIs.
* rCUDA: Reducing the Number of GPU-Based Accelerators in High Performance Clusters (HPCS 2010) \[[Paper](https://ieeexplore.ieee.org/document/5547126)] \[[Homepage](http://www.rcuda.net/)]
  * Universidad Politécnica de Valencia & Universidad Jaume I
  * Remote GPU acceleration; HPC clusters.

## Industrial Products

* VMware vSphere Bitfusion \[[Docs](https://docs.vmware.com/en/VMware-vSphere-Bitfusion/index.html)]
* VirtAI Tech Orion \[[GitHub](https://github.com/virtaitech/orion) (in Chinese)]

## Acronyms

* RPC: Remote Procedure Call


# Memory Disaggregation

## CXL-based Disaggregation

* CXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation for Billion-Scale Approximate Nearest Neighbor Search ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/jang)]
  * KAIST & Panmnesia, Inc.
  * Approximate nearest neighbor search (ANNS) services.
* Overcoming the Memory Wall with CXL-Enabled SSDs ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/yang-shao-peng)]
  * Syracuse & DGIST & FADU Inc. & Soongsil
  * CXL-flash: CXL-enabled *flash device*; caching and prefetching.
* TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory ([ASPLOS 2023](/reading-notes/conference/asplos-2023)) \[[Personal Notes](/reading-notes/conference/asplos-2023/tpp)] \[[Paper](https://dl.acm.org/doi/10.1145/3582016.3582063)] \[[Code](https://lwn.net/Articles/876993/)]
  * UMich SymbioticLab & NVIDIA & Meta
  * CXL 1.1
  * Identify and place hot/cold pages to appropriate memory tiers (i.e., local memory or CXL memory).
* Pond: CXL-Based Memory Pooling Systems for Cloud Platforms ([ASPLOS 2023](/reading-notes/conference/asplos-2023)) \[[Paper](https://dl.acm.org/doi/abs/10.1145/3575693.3578835)]
  * Microsoft Azure
* Direct Access, High-Performance Memory Disaggregation with DirectCXL ([ATC 2022](/reading-notes/conference/atc-2022)) \[[Personal Notes](/reading-notes/conference/atc-2022/directcxl)] \[[Paper](https://www.usenix.org/conference/atc22/presentation/gouk)]
  * KAIST
  * CXL 2.0
  * 6.2x shorter latency & 3x better performance than RDMA-based memory disaggregation.

## RDMA-based Disaggregation

* DisaggRec: Architecting Disaggregated Systems for Large-Scale Personalized Recommendation (arXiv 2212.00939) \[[Personal Notes](broken://pages/PZK9r9RRQdKqiSGt6JW4)] \[[Paper](https://arxiv.org/abs/2212.00939)]
  * Meta AI & WashU & UPenn & Cornell & Intel
  * Deep learning recommendation models; partition *embedding tables*.
* Hydra: Resilient and Highly Available Remote Memory (FAST 2022) \[[Paper](https://www.usenix.org/conference/fast22/presentation/lee)] \[[Code](https://github.com/SymbioticLab/Hydra)]
  * UMich SymbioticLab
  * In-memory erasure coding.
* Rethinking Software Runtimes for Disaggregated Memory (ASPLOS 2021) \[[Paper](https://dl.acm.org/doi/10.1145/3445814.3446713)] \[[Code](https://github.com/project-kona/asplos21-ae)]
  * VMWare & Penn State & ETH & EPFL & UMich & Google
  * Kona
  * Cache coherence.
* Effectively Prefetching Remote Memory with Leap ([ATC 2020](/reading-notes/conference/atc-2020)) \[[Paper](https://www.usenix.org/conference/atc20/presentation/al-maruf)] \[[Code](https://github.com/SymbioticLab/leap)]
  * UMich SymbioticLab
  * **Best Paper**
  * Memory prefetching
* LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation ([OSDI 2018](/reading-notes/conference/osdi-2018)) \[[Paper](https://www.usenix.org/conference/osdi18/presentation/shan)] \[[Code](https://github.com/WukLab/LegoOS)]
  * Purdue
  * **Best Paper**
  * *Splitkernel*
* Efficient Memory Disaggregation with Infiniswap (NSDI 2017) \[[Paper](https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/gu)] \[[Code](https://github.com/SymbioticLab/Infiniswap)]
  * UMich SymbioticLab
  * Remote memory paging system.


# Resource Fragmentation

## GPU Fragmentation

* Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent ([ATC 2023](/reading-notes/conference/atc-2023)) \[[Paper](https://www.usenix.org/conference/atc23/presentation/weng)] \[[Code](https://github.com/hkust-adsl/kubernetes-scheduler-simulator)]
  * HKUST & Alibaba
  * Quantify GPU fragmentation in GPU-sharing clusters.
  * Guided scheduling with *fragmentation gradient descent.*
* HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees ([OSDI 2020](/reading-notes/conference/osdi-2020)) \[[Personal Notes](/reading-notes/conference/osdi-2020/hived)] \[[Paper](https://www.usenix.org/conference/osdi20/presentation/zhao-hanyu)] \[[Code](https://github.com/microsoft/hivedscheduler)]
  * PKU & HKU & MSRA
  * Consider GPU affinity; resource reservation.

## General Fragmentation

* Large-scale cluster management at Google with Borg (EuroSys 2015) \[[Paper](https://dl.acm.org/doi/10.1145/2741948.2741964)]
  * Google
  * Reduce *stranded resources* that cannot be used because another resource on the machine is fully allocated.
* Multi-Resource Packing for Cluster Schedulers (SIGCOMM 2014) \[[Paper](https://dl.acm.org/doi/10.1145/2619239.2626334)]
  * Microsoft
  * Tetris: *Pack* jobs to avoid resource fragmentation and over-allocation.
* Evaluating job packing in warehouse-scale computing ([CLUSTER 2014](https://github.com/mental2008/awesome-papers/blob/develop/reading-notes/conference/CLUSTER-2014/README.md)) \[[Personal Notes](/reading-notes/conference/cluster-2014/evaluating-job-packing)] \[[Paper](https://static.googleusercontent.com/media/research.google.com/en/pubs/archive/43103.pdf)]
  * Google
  * **Four** metrics for evaluating the packing efficiency of schedulers: *aggregate utilization*, *hole filling*, *workload inflation*, and *cluster compaction*.


# Cloud Computing

## Technical Reports

* Above the Clouds: A Berkeley View of Cloud Computing (Technical Report No. UCB/EECS-2009-28) \[[Paper](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html)]
  * UC Berkeley

## Detailed Aspects

* [Sky Computing](/paper-list/cloud-computing/sky-computing)


# Sky Computing

* SkyPilot: An Intercloud Broker for Sky Computing ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/yang-zongheng)] \[[Code](https://github.com/skypilot-org/skypilot)] \[[Docs](https://skypilot.readthedocs.io/en/latest/)]
  * UC Berkeley
  * Cherry-picking clouds.
* Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlays ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Personal Notes](/reading-notes/conference/nsdi-2023/skyplane)] \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/jain)] \[[Code](https://github.com/skyplane-project/skyplane)] \[[Docs](https://skyplane.org/en/latest/)]
  * UC Berkeley
  * *Bulk data transfer* between *cloud object stores*.
* The Sky Above The Clouds: A Berkeley View on the Future of Cloud Computing (arXiv 2205.07147) \[[Paper](https://arxiv.org/abs/2205.07147)]
  * UC Berkeley
* From Cloud Computing to Sky Computing ([HotOS 2021](/reading-notes/conference/hotos-2021)) \[[Personal Notes](/reading-notes/conference/hotos-2021/sky-computing)] \[[Paper](https://dl.acm.org/doi/10.1145/3458336.3465301)]
  * UC Berkeley


# Serverless Computing

* Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider ([ATC 2020](/reading-notes/conference/atc-2020)) \[[Personal Notes](/reading-notes/conference/atc-2020/serverless-in-the-wild-characterizing-and-optimizing-the-serverless-workload)] \[[Paper](https://www.usenix.org/conference/atc20/presentation/shahrad)] \[[Trace](https://github.com/Azure/AzurePublicDataset)]
  * Microsoft Azure & Microsoft Research
  * **Trace analysis**; a policy for *reducing the number of cold start function executions*.
* Cloud Programming Simplified: A Berkeley View on Serverless Computing (Technical Report No. UCB/EECS-2019-3) \[[Paper](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.html)]
  * UC Berkeley


# Spot Instances

* SpotServe: Serving Generative Large Language Models on Preemptible Instances ([ASPLOS 2024](/reading-notes/conference/asplos-2024)) \[[Personal Notes](/reading-notes/conference/asplos-2024/spotserve)] \[[Paper](https://arxiv.org/abs/2311.15566)] \[[Code](https://github.com/Hsword/SpotServe)]
  * CMU & PKU & CUHK
* Can't Be Late: Optimizing Spot Instance Savings under Deadlines ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/wu-zhanghao)] \[[Trace](https://github.com/skypilot-org/spot-traces)]
  * UC Berkeley
  * **Outstanding Paper**
* Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances ([NSDI 2024](/reading-notes/conference/nsdi-2024)) \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/duan)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-duan.pdf)] \[[Code](https://github.com/JF-D/Parcae)]
  * CUHK & ByteDance & CMU & UCLA & Microsoft


# Remote Direct Memory Access (RDMA)

## General RDMA

* X-RDMA: Effective RDMA Middleware in Large-scale Production Environments ([CLUSTER 2019](/reading-notes/conference/cluster-2019)) \[[Paper](https://ieeexplore.ieee.org/document/8891004)]
  * Alibaba
  * Focus on robustness, scalability, and maintainability.
* FreeFlow: Software-based Virtual RDMA Networking for Containerized Clouds ([NSDI 2019](/reading-notes/conference/nsdi-2019)) \[[Paper](https://www.usenix.org/conference/nsdi19/presentation/kim)] \[[Code](https://github.com/Microsoft/Freeflow)]
  * CMU & Microsoft & Alibaba & ByteDance
  * A software-based RDMA virtualization framework designed for containerized clouds.
* Revisiting Network Support for RDMA ([SIGCOMM 2018](/reading-notes/conference/sigcomm-2018)) \[[Personal Notes](/reading-notes/conference/sigcomm-2018/irn)] \[[Paper](https://dl.acm.org/doi/10.1145/3230543.3230557)]
  * UC Berkeley & ICSI & Mellanox & NYU & UW
  * IRN: Better handling of packet losses; eliminate the need for PFC.
* RDMA over Commodity Ethernet at Scale (SIGCOMM 2016) \[[Paper](https://dl.acm.org/doi/10.1145/2934872.2934908)]
  * Microsoft
  * Challenges using *RoCEv2*; a DSCP (Differentiated Services Code Point) based PFC mechanism.
* Congestion Control for Large-Scale RDMA Deployments (SIGCOMM 2015) \[[Paper](https://dl.acm.org/doi/10.1145/2785956.2787484)]
  * Microsoft & Mellanox & UCSB
  * DCQCN: A congestion control scheme for *RoCEv2*, to alleviate the problems of PFC.

## RDMA for Deep Learning

* Fast Distributed Deep Learning over RDMA (EuroSys 2019) \[[Paper](https://dl.acm.org/doi/10.1145/3302424.3303975)]
  * MSRA
* Towards Zero Copy Dataflows using RDMA (SIGCOMM 2017 Posters and Demos) \[[Paper](https://dl.acm.org/doi/10.1145/3123878.3131975)] \[[Code](https://github.com/tensorflow/networking/tree/master/tensorflow_networking/gdr)]
  * HKUST
  * Merged into TensorFlow.

## RDMA for Storage

* Empowering Azure Storage with RDMA ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/bai)]
  * Microsoft
  * Production experience in Microsoft Azure
  * Around **70%** of traffic in Azure is RDM&#x41;**.**
* When Cloud Storage Meets RDMA ([NSDI 2021](/reading-notes/conference/nsdi-2021)) \[[Paper](https://www.usenix.org/conference/nsdi21/presentation/gao)]
  * NJU & Alibaba
  * Pangu
  * Production experience in Alibaba Cloud
  * Two workarounds to handle PFC storms: *shutdown, RDMA/TCP switching*.

## Performance Isolation

* Understanding RDMA Microarchitecture Resources for Performance Isolation ([NSDI 2023](/reading-notes/conference/nsdi-2023)) \[[Personal Notes](/reading-notes/conference/nsdi-2023/husky)] \[[Paper](https://www.usenix.org/conference/nsdi23/presentation/kong)] \[[Benchmark Suite](https://github.com/host-bench/husky)]
  * Duke & Microsoft & SJTU
  * Develop a *test suite* to *evaluate* RDMA performance isolation solutions.

## Acronyms

* PFC: Priority Flow Control
* RoCE: RDMA over Converged Ethernet
* IBoE: InfiniBand over Ethernet


# Research Skills

## Generally Speaking

* [How to Have a Bad Career in Research/Academia](https://people.eecs.berkeley.edu/~pattrsn/talks/BadCareer.pdf), by David A. Patterson
* [You and Your Research](https://blog.samaltman.com/you-and-your-research), by Richard Hamming
* [Principles of Effective Research](https://michaelnielsen.org/blog/principles-of-effective-research/), by Michael A. Nielsen
* [Research as a Stochastic Decision Process](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html), by Jacob Steinhardt

## Before you start your Ph.D.

* [Ph.D. Advice: Useful Thoughts about Research](https://www.eecs.harvard.edu/htk/phdadvice/), by H.T. Kung
* The Ph.D. Grind, by Philip J. Guo
* [A Survival Guide to a PhD](http://karpathy.github.io/2016/09/07/phd/), by Andrej Karpathy
* [A PhD Is Not Enough!: A Guide to Survival in Science](https://www.usu.edu/biomath/files/peter-feibelm.pdf), by Peter J. Feibelman

## Paper Reading

* [How to Read a Paper](http://ccr.sigcomm.org/online/files/p83-keshavA.pdf), by Srinivasan Keshav \[[Personal Note](/reading-notes/journal/ccr/2007/how-to-read-a-paper)]
* [Efficient Reading of Papers in Science and Technology](https://www.cs.columbia.edu/~hgs/netbib/efficientReading.pdf), by Michael J. Hanson and Dylan J. McNamee

## Performance Evaluation

* [Always Measure One Level Deeper](https://cacm.acm.org/research/always-measure-one-level-deeper/), by John Ousterhout
* [Performance Hints](https://abseil.io/fast/hints.html), by Jeff Dean and Sanjay Ghemawat

## Writing a Paper

* [How (and How Not) to Write a Good Systems Paper](https://www.usenix.org/legacy/publications/library/proceedings/dsl97/good_paper.html), by Roy Levin and David D. Redell (9th SOSP Program Committee Co-chairmen)
* [The Science of Scientific Writing](https://www.usenix.org/sites/default/files/gopen_and_swan_science_of_scientific_writing.pdf), by George D. Gopen and Judith A. Swan
* [How to write a technical paper or a research paper](https://homes.cs.washington.edu/~mernst/advice/write-technical-paper.html), by Michael Ernst
* [Writing Perfect Papers](https://iqua.ece.toronto.edu/papers/writing-perfect-papers-2021.pdf), by Baochun Li
* [10 tips on how we write papers](https://www.cell.com/matter/fulltext/S2590-2385\(22\)00543-4), by Edward H. Sargent

## Presentation

* [The Art of Presentation](https://www.cs.cityu.edu.hk/~jia/research/the-art-of-presentation.pdf), by Baochun Li

## Faculty Job Search

* [Philip’s notes on the tenure-track assistant professor job search](https://xb00dx.github.io/files/guo-faculty-job-search.pdf), by Philip J. Guo

## Student Mentorship

* [Your students are your legacy](https://dl.acm.org/doi/10.1145/1467247.1467259), by David A. Patterson \[[中文译本](https://blog.sciencenet.cn/blog-414166-302397.html)]

***

## Distributed Systems

* [Software Engineering Advice from Building Large-Scale Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf), by Jeff Dean
* [Designs, Lessons and Advice from Building Large Distributed Systems](https://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf), by Jeff Dean

## AI Research

* [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html), by Richard S. Sutton
* [An Opinionated Guide to ML Research](http://joschu.net/blog/opinionated-guide-ml-research.html), by John Schulman
* [Practicing AI research](https://www.jasonwei.net/blog/practicing-ai-research), by Jason Wei

***

## 读博心路历程

* [博士这五年](https://zhuanlan.zhihu.com/p/25099638), by 李沐
* [机器学习科研的十年](https://zhuanlan.zhihu.com/p/74249758), by 陈天奇
* [博士五年总结系列](https://yuandong-tian.com/five_year_summary_of_PhD.pdf), by 田渊栋


# Miscellaneous

## Career

* [How To Be Successful](https://blog.samaltman.com/how-to-be-successful), by Sam Altman


# Conference

## 2026

|                       Conference                       |        When        |                          Where                          | Remarks                                                               |
| :----------------------------------------------------: | :----------------: | :-----------------------------------------------------: | --------------------------------------------------------------------- |
|    [SOSP 2026](/reading-notes/conference/sosp-2026)    | Sep 29-Oct 2, 2026 |      Clarion Congress Hotel Prague, Prague, Czechia     | Incoming                                                              |
|    [OSDI 2026](/reading-notes/conference/osdi-2026)    |   Jul 13-15, 2026  |                     Seattle, WA, USA                    | 🧐 Incoming                                                           |
|    [ICML 2026](/reading-notes/conference/icml-2026)    |   Jul 6-11, 2026   | COEX Convention & Exhibition Center, Seoul, South Korea | 🧐                                                                    |
|    [ISCA 2026](/reading-notes/conference/isca-2026)    | Jun 27-Jul 1, 2026 |       Raleigh Convention Center, Raleigh, NC, USA       | Incoming                                                              |
|    [CAIS 2026](/reading-notes/conference/cais-2026)    |   May 26-29, 2026  |                    San Jose, CA, USA                    | 🧐                                                                    |
|   [MLSys 2026](/reading-notes/conference/mlsys-2026)   |   May 18-22, 2026  |        Hyatt Regency Bellevue, Bellevue, WA, USA        | 🧐                                                                    |
|    [NSDI 2026](/reading-notes/conference/nsdi-2026)    |    May 4-6, 2026   |      Hyatt Regency Lake Washington, Renton, WA, USA     | 🧐                                                                    |
| [EuroSys 2026](/reading-notes/conference/eurosys-2026) |   Apr 27-30, 2026  |                      Edinburgh, UK                      | 😎                                                                    |
|  [ASPLOS 2026](/reading-notes/conference/asplos-2026)  |   Mar 22-26, 2026  |                     Pittsburgh, USA                     | 🧐                                                                    |
|    [FAST 2026](/reading-notes/conference/fast-2026)    |   Feb 24-26, 2026  |     Hyatt Regency Santa Clara, Santa Clara, CA, USA     |                                                                       |
|    [HPCA 2026](/reading-notes/conference/hpca-2026)    | Jan 31-Feb 4, 2026 |                    Sydney, Australia                    | 🧐 Co-located with [PPoPP 2026](/reading-notes/conference/ppopp-2026) |
|   [PPoPP 2026](/reading-notes/conference/ppopp-2026)   | Jan 31-Feb 4, 2026 |                    Sydney, Australia                    | 🧐 Co-located with [HPCA 2026](/reading-notes/conference/hpca-2026)   |

## 2025

|                       Conference                       |        When        |                    Where                    | Remarks                                                                                                                                                     |
| :----------------------------------------------------: | :----------------: | :-----------------------------------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|      [SC 2025](/reading-notes/conference/sc-2025)      |   Nov 16-21, 2025  |      St. Louis, Missouri, United States     |                                                                                                                                                             |
|    [SOSP 2025](/reading-notes/conference/sosp-2025)    |   Oct 13-16, 2025  | Lotte Hotel World, Seoul, Republic of Korea | 🧐                                                                                                                                                          |
| [SIGCOMM 2025](/reading-notes/conference/sigcomm-2025) |   Sep 8-11, 2025   |              Coimbra, Portugal              | 🧐                                                                                                                                                          |
|    [ICML 2025](/reading-notes/conference/icml-2025)    |   Jul 13-19, 2025  |     Vancouver Convention Center, Canada     |                                                                                                                                                             |
|     [ATC 2025](/reading-notes/conference/atc-2025)     |    Jul 7-9, 2025   |    Sheraton Boston Hotel, Boston, MA, USA   | 🧐😎 Co-located with [OSDI 2025](/reading-notes/conference/osdi-2025)                                                                                       |
|    [OSDI 2025](/reading-notes/conference/osdi-2025)    |    Jul 7-9, 2025   |    Sheraton Boston Hotel, Boston, MA, USA   | 🧐😎 Co-located with [ATC 2025](/reading-notes/conference/atc-2025)                                                                                         |
|   [HotOS 2025](/reading-notes/conference/hotos-2025)   |   May 14-16, 2025  |            Banff, Alberta, Canada           | 🧐                                                                                                                                                          |
|   [MLSys 2025](/reading-notes/conference/mlsys-2025)   |   May 12-15, 2025  |    Santa Clara Convention Center, CA, USA   | 🧐                                                                                                                                                          |
|    [NSDI 2025](/reading-notes/conference/nsdi-2025)    |   Apr 28-30, 2025  |            Philadelphia, PA, USA            | 🧐😎                                                                                                                                                        |
|  [ASPLOS 2025](/reading-notes/conference/asplos-2025)  | Mar 30-Apr 3, 2025 |            Rotterdam, Netherlands           | 🧐 Co-located with [EuroSys 2025](/reading-notes/conference/eurosys-2025)                                                                                   |
| [EuroSys 2025](/reading-notes/conference/eurosys-2025) | Mar 30-Apr 3, 2025 |            Rotterdam, Netherlands           | 🧐 Co-located with [ASPLOS 2025](/reading-notes/conference/asplos-2025)                                                                                     |
|    [HPCA 2025](/reading-notes/conference/hpca-2025)    |    Mar 1-5, 2025   |              Las Vegas, NV, USA             | 🧐 Co-located with [PPoPP 2025](/reading-notes/conference/ppopp-2025), [CGO 2025](https://2025.cgo.org), [CC 2025](https://conf.researchr.org/home/CC-2025) |
|   [PPoPP 2025](/reading-notes/conference/ppopp-2025)   |    Mar 1-5, 2025   |              Las Vegas, NV, USA             | 🧐 Co-located with [HPCA 2025](/reading-notes/conference/hpca-2025), [CGO 2025](https://2025.cgo.org), [CC 2025](https://conf.researchr.org/home/CC-2025)   |

## 2024

|                       Conference                       |        When        |                          Where                         | Remarks                                                             |
| :----------------------------------------------------: | :----------------: | :----------------------------------------------------: | ------------------------------------------------------------------- |
| [NeurIPS 2024](/reading-notes/conference/neurips-2024) |   Dec 10-15, 2024  |           Vancouver Convention Center, Canada          | 🧐                                                                  |
|    [SoCC 2024](/reading-notes/conference/socc-2024)    |   Nov 22-24, 2024  |                Seattle, Washington, USA                | 🧐                                                                  |
| [HotNets 2024](/reading-notes/conference/hotnets-2024) |   Nov 18-19, 2024  |                 Irvine, California, USA                | 🧐                                                                  |
|      [SC 2024](/reading-notes/conference/sc-2024)      |   Nov 17-22, 2024  |                    Atlanta, GA, USA                    | 🧐                                                                  |
|    [SOSP 2024](/reading-notes/conference/sosp-2024)    |    Nov 4-6, 2024   |                Hilton Austin, Texas, USA               | 🧐                                                                  |
|    [VLDB 2024](/reading-notes/conference/vldb-2024)    |   Aug 26-30, 2024  |                    Guangzhou, China                    | 🧐                                                                  |
| [SIGCOMM 2024](/reading-notes/conference/sigcomm-2024) |    Aug 4-8, 2024   |                    Sydney, Australia                   | 🧐                                                                  |
|    [ICML 2024](/reading-notes/conference/icml-2024)    |   Jul 21-27, 2024  | Messe Wien Exhibition Congress Center, Vienna, Austria |                                                                     |
|     [ATC 2024](/reading-notes/conference/atc-2024)     |   Jul 10-12, 2024  |                  Santa Clara, CA, USA                  | 🧐 Co-located with [OSDI 2024](/reading-notes/conference/osdi-2024) |
|    [OSDI 2024](/reading-notes/conference/osdi-2024)    |   Jul 10-12, 2024  |                  Santa Clara, CA, USA                  | 🧐 Co-located with [ATC 2024](/reading-notes/conference/atc-2024)   |
|    [ISCA 2024](/reading-notes/conference/isca-2024)    | Jun 29-Jul 3, 2024 |                 Buenos Aires, Argentina                | 🧐                                                                  |
|    [CVPR 2024](/reading-notes/conference/cvpr-2024)    |   Jun 17-21, 2024  |       Seattle Convention Center, Seattle, WA, USA      | 🧐                                                                  |
|   [MLSys 2024](/reading-notes/conference/mlsys-2024)   |   May 13-16, 2024  |           Santa Clara Convention Center, USA           | 🧐                                                                  |
|  [ASPLOS 2024](/reading-notes/conference/asplos-2024)  | Apr 27-May 1, 2024 |      Hilton La Jolla Torrey Pines, San Diego, USA      | 🧐                                                                  |
| [EuroSys 2024](/reading-notes/conference/eurosys-2024) |   Apr 23-26, 2024  |                     Athens, Greece                     |                                                                     |
|    [NSDI 2024](/reading-notes/conference/nsdi-2024)    |   Apr 16-18, 2024  |                  Santa Clara, CA, USA                  | 🧐                                                                  |
|                        HPCA 2024                       |    Mar 2-6, 2024   |                 Edinburgh, Scotland, UK                |                                                                     |

## 2023

|                        Conference                        |        When        |             Where             | Remarks                                                             |
| :------------------------------------------------------: | :----------------: | :---------------------------: | ------------------------------------------------------------------- |
|  [NeurIPS 2023](/reading-notes/conference/neurips-2023)  |   Dec 10-16, 2023  |  New Orleans, Louisiana, USA  |                                                                     |
|       [SC 2023](/reading-notes/conference/sc-2023)       |   Nov 12-17, 2023  |     Denver, Colorado, USA     | 🧐                                                                  |
|     [SoCC 2023](/reading-notes/conference/socc-2023)     | Oct 30-Nov 1, 2023 |  Santa Cruz, California, USA  | 🧐                                                                  |
|     [SOSP 2023](/reading-notes/conference/sosp-2023)     |   Oct 23-26, 2023  |        Koblenz, Germany       | 🧐                                                                  |
|  [SIGCOMM 2023](/reading-notes/conference/sigcomm-2023)  |   Sep 10-14, 2023  |         New York, USA         |                                                                     |
| [HotChips 2023](/reading-notes/conference/hotchips-2023) |   Aug 27-29, 2023  |  Hybrid, Stanford University  |                                                                     |
|     [ICML 2023](/reading-notes/conference/icml-2023)     |   Jul 23-29, 2023  |    Hawaii Convention Center   |                                                                     |
|      [ATC 2023](/reading-notes/conference/atc-2023)      |   Jul 10-12, 2023  |        Boston, MA, USA        | 😎 Co-located with [OSDI 2023](/reading-notes/conference/osdi-2023) |
|     [OSDI 2023](/reading-notes/conference/osdi-2023)     |   Jul 10-12, 2023  |        Boston, MA, USA        | 😎 Co-located with [ATC 2023](/reading-notes/conference/atc-2023)   |
|    [HotOS 2023](/reading-notes/conference/hotos-2023)    |   Jun 22-24, 2023  | Providence, Rhode Island, USA |                                                                     |
|   [SIGMOD 2023](/reading-notes/conference/sigmod-2023)   |   Jun 18-23, 2023  |        Seattle, WA, USA       |                                                                     |
|     [ISCA 2023](/reading-notes/conference/isca-2023)     |   Jun 17-21, 2023  |        Orlando, FL, USA       |                                                                     |
|    [MLSys 2023](/reading-notes/conference/mlsys-2023)    |    Jun 4-8, 2023   |     Southern Florida, USA     |                                                                     |
|  [EuroSys 2023](/reading-notes/conference/eurosys-2023)  |   May 9-12, 2023   |          Rome, Italy          |                                                                     |
|     [NSDI 2023](/reading-notes/conference/nsdi-2023)     |   Apr 17-19, 2023  |        Boston, MA, USA        |                                                                     |
|   [ASPLOS 2023](/reading-notes/conference/asplos-2023)   |   Mar 25-29, 2023  |       Vancouver, Canada       |                                                                     |

## 2022

|                       Conference                       |        When        |                     Where                     | Remarks |
| :----------------------------------------------------: | :----------------: | :-------------------------------------------: | ------- |
|      [SC 2022](/reading-notes/conference/sc-2022)      |   Nov 13-18, 2022  | Kay Bailey Hutchison Convention Center Dallas |         |
|    [SoCC 2022](/reading-notes/conference/socc-2022)    |   Nov 7-11, 2022   |             San Francisco, CA, USA            | 👨‍💻   |
| [SIGCOMM 2022](/reading-notes/conference/sigcomm-2022) |   Aug 22-26, 2022  |             Amsterdam, Netherlands            |         |
|     [ATC 2022](/reading-notes/conference/atc-2022)     |   Jul 11-13, 2022  |               Carlsbad, CA, USA               |         |
|    [OSDI 2022](/reading-notes/conference/osdi-2022)    |   Jul 11-13, 2022  |               Carlsbad, CA, USA               |         |
|   [IPDPS 2022](/reading-notes/conference/ipdps-2022)   | May 30-Jun 3, 2022 |                    Virtual                    |         |
| [EuroSys 2022](/reading-notes/conference/eurosys-2022) |    Apr 5-8, 2022   |                 Rennes, France                |         |
|    [NSDI 2022](/reading-notes/conference/nsdi-2022)    |    Apr 4-6, 2022   |                Renton, WA, USA                |         |

## 2021

|                       Conference                       |       When      |  Where  | Remarks |
| :----------------------------------------------------: | :-------------: | :-----: | ------- |
|    [SoCC 2021](/reading-notes/conference/socc-2021)    |  Nov 1-4, 2021  | Virtual | 👨‍💻   |
|    [SOSP 2021](/reading-notes/conference/sosp-2021)    | Oct 25-29, 2021 | Virtual | 👨‍💻   |
|     [ATC 2021](/reading-notes/conference/atc-2021)     | Jul 14–16, 2021 | Virtual |         |
| [EuroSys 2021](/reading-notes/conference/eurosys-2021) | Apr 26-28, 2021 | Virtual |         |
|    [NSDI 2021](/reading-notes/conference/nsdi-2021)    | Apr 12-14, 2021 | Virtual |         |

## 2020

|                       Conference                       |       When      |      Where      | Remarks |
| :----------------------------------------------------: | :-------------: | :-------------: | ------- |
|    [OSDI 2020](/reading-notes/conference/osdi-2020)    |  Nov 4-6, 2020  |     Virtual     |         |
|     [ATC 2020](/reading-notes/conference/atc-2020)     | Jul 15-17, 2020 |     Virtual     |         |
| [EuroSys 2020](/reading-notes/conference/eurosys-2020) | Apr 27-30, 2020 |     Virtual     |         |
|  [ASPLOS 2020](/reading-notes/conference/asplos-2020)  | Mar 16-20, 2020 |     Virtual     |         |
|   [MLSys 2020](/reading-notes/conference/mlsys-2020)   |  Mar 2-4, 2020  | Austin, TX, USA |         |

## 2019

|                       Conference                       |       When      |             Where            | Remarks |
| :----------------------------------------------------: | :-------------: | :--------------------------: | ------- |
| [CLUSTER 2019](/reading-notes/conference/cluster-2019) | Sep 23-26, 2019 | Albuquerque, New Mexico, USA |         |
| [EuroSys 2019](/reading-notes/conference/eurosys-2019) | Mar 25-28, 2019 |       Dresden, Germany       |         |
|    [NSDI 2019](/reading-notes/conference/nsdi-2019)    | Feb 26-28, 2019 |        Boston, MA, USA       |         |

## 2018

|                       Conference                       |       When      |       Where       | Remarks |
| :----------------------------------------------------: | :-------------: | :---------------: | ------- |
| [SIGCOMM 2018](/reading-notes/conference/sigcomm-2018) | Aug 20-25, 2018 | Budapest, Hungary |         |

## 2017

|                      Conference                      |       When      |             Where            | Remarks |
| :--------------------------------------------------: | :-------------: | :--------------------------: | ------- |
|   [SoCC 2017](/reading-notes/conference/socc-2017)   | Sep 25-27, 2017 | Santa Clara, California, USA |         |
| [ASPLOS 2017](/reading-notes/conference/asplos-2017) |  Apr 8-12, 2017 |         Xi'an, China         |         |
|   [NSDI 2017](/reading-notes/conference/nsdi-2017)   | Mar 27-29, 2017 |        Boston, MA, USA       |         |

## Notes

* 😎: In-person attendance.
* 👨‍💻: Virtual attendance.
* 🧐: Papers are organized.


# SOSP 2026

## Meta Info

Homepage: <https://sigops.org/s/conferences/sosp/2026/>

Paper list: <https://sigops.org/s/conferences/sosp/2026/accepted.html>

### Acceptance Rate

15.9% (= 62 / 390)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Training Emulation
    * A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with CrystalLLM
      * Alibaba & SJTU & Harvard & ZJU
  * RL Post-Training
    * Scaling Weight Transfer in LLM RL Training with Reference-Oriented Storage
      * UW-Madison & ByteDance Seed
* LLM Inference
  * Deterministic Inference
    * Axiom: Achieving Determinism in LLM Inference
      * MSR & UW
  * Offline Inference
    * Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
      * SJTU & Alibaba
  * MoE Inference
    * Democratizing MoE LLM Decoding via Barrier-Free Expert Parallelism
      * USC & Google DeepMind & UT Austin
  * Multi-LLM Serving
    * Multi-LLM Serving at Production Scale \[[Code](https://github.com/Janus2026/Janus)]
      * PKU & JD & UCAS & USTC & SJTU
  * Workload Characterization
    * LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems
      * SCITIX & THU
  * End-to-End Serving
    * Taming Inference Workloads at Global Scale: Foundation Model Serving in SystemX
      * AWS
* LLM Applications
  * Agentic Systems
    * Skill VM: Write Once, Run Everywhere Efficiently \[[arXiv](https://arxiv.org/abs/2604.03088)] \[[Code](https://github.com/SJTU-IPADS/SkVM/)] \[[Homepage](https://skillvm.ai/)]
      * SJTU IPADS
      * Introduce **SkVM**, treating agent skills as programs and LLMs plus harnesses as heterogeneous processors to improve skill portability across models and agent runtimes.
      * Combine AOT compilation based on model-harness capability profiling with runtime JIT optimization, including code solidification and feedback-driven skill rewriting.
      * Evaluate across eight LLMs and three agent harnesses; improve task completion by about 10%, reduce token cost by up to 40%, and cut end-to-end latency by 19-50x through JIT-boost code solidification.
  * Agent Data Infrastructure
    * Supporting Agents on Streaming Data via a Forkable Shared Log
      * UIUC
  * Voice Applications
    * Scalable Context Orchestration for Serving LLMs Over Voice
      * SJTU & AgenticSys Labs

### Diffusion Models

* Diffusion Model Serving
  * DiFlow: A System for Micro-Serving Text-to-Image Diffusion Workflows \[[arXiv](https://arxiv.org/abs/2604.08123)]
    * HKUST & Alibaba
    * **Our work!**
    * Decomposes text-to-image diffusion workflows into independently managed model-execution nodes.
    * Enables per-model scaling, cross-workflow model sharing, and adaptive model parallelism for bursty serving workloads.

### Model Serving

* Multi-Model Serving
  * SANDHI: Fine-grained Merging for Memory Efficient Multi-Model Serving
    * GaTech & Microsoft
* Trusted Inference
  * Cardamom: A Multi-GPU Framework for TB-Scale Encrypted Inference
    * CMU & UT Austin & NVIDIA

### Deep Learning Compilation

* Agent-Based Kernel Generation
  * Guiding Agentic GPU Kernel Optimization with Data Flow Invariants \[[arXiv](https://arxiv.org/abs/2604.18616)]
    * HKUST & UCAS & UC Riverside & USTC & XJTU & THU & Stanford
* GPU Kernels and Compilers
  * Taming Dynamism on GPUs: Cross-SM Kernel Fusion via SM Cooperation and Just-in-Time Reduction
    * SJTU IPADS
    * Introduce **MorphKernel**, targeting dynamic and imbalanced GPU operators such as decoding attention and MoE expert activation.
    * Fuse multi-phase kernels across SMs by decomposing coarse phase barriers into sub-task dependencies with fine-grained SM synchronization and on-chip data exchange, reducing HBM traffic for intermediate results.
    * Provide a declarative programming abstraction over operators such as map, iter, and reduce, while the runtime dynamically decides scheduling, communication, and parallel partitioning from input shapes.
    * Use TMA-based send/recv primitives, GPC-topology-aware communication, and per-SM instruction queues; improve performance by 1.3x on average over real workloads.
* Kernel Verification
  * M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification
    * Penn State & ICT, CAS & PKU
* Wafer-Scale Compilation and Runtime
  * MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference
    * Edinburgh & MSR
  * Wavel: A Fast and Efficient Compilation System for Wafer-Scale Accelerators
    * Edinburgh & MSR

### Resource Management

* GPU Scheduling and Sharing
  * Beyond Utilization: Energy-Conscious GPU Sharing for Inference Serving
    * UT Austin
  * Efficient GPU Multitasking with Morphable Kernels
    * THU
  * Linux AGX: An Adaptive GPU eXtension to Linux Fair Scheduling for Physical AI and Robotic Systems
    * UC Riverside

### Model Storage and Data Pipeline

* Model Storage
  * TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models
    * UVA & Harvard

### ML for Systems

* System Optimization
  * TuxBot: Semantic-Aware Online OS Tuning with LLMs
    * Columbia & IBM Research


# OSDI 2026

## Meta Info

Homepage: <https://www.usenix.org/conference/osdi26>

Paper list: <https://www.usenix.org/conference/osdi26/technical-sessions>

### Acceptance Rate

20.0% (= 136 / 681)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * RL Post-Training
    * RollArt: Disaggregated Multi-Task Agentic RL Training at Scale \[[Paper](https://www.usenix.org/conference/osdi26/presentation/gao)]
      * HKUST & Alibaba & Tongyi Lab
      * Propose **RollArt**, mapping agentic RL pipeline stages to best-fit hardware and coordinating prefill, decoding, environment, and reward workloads.
    * Weave: Efficient Co-Scheduling for Disaggregated RL Post-Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wu-tianyuan)]
      * HKUST & UIUC & Alibaba
      * Present **Weave**, a cross-cluster scheduler that reclaims dependency bubbles between rollout and training phases in disaggregated RL post-training.
    * RLinf: Flexible and Efficient Large-Scale Reinforcement Learning via Macro-to-Micro Flow Transformation \[[Paper](https://www.usenix.org/conference/osdi26/presentation/yu-chao)]
      * THU & Infinigence-AI & PKU & UC Berkeley & Zhongguancun Academy & BUAA & SJTU
      * Introduce **RLinf**, transforming RL workflows from macro pipelines into micro flows to improve scheduling flexibility and hardware utilization.
    * DynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wang-yuanqing)]
      * PKU & Infinigence-AI & ICT, CAS & BUAA & THU & SJTU
      * Present **DynaRL**, a dynamic scheduler that reallocates compute, memory, and communication resources across heterogeneous RL components.
    * Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning \[[Paper](https://www.usenix.org/conference/osdi26/presentation/qin)]
      * THU & Moonshot AI
      * Introduce **Seer**, using prompt-level context learning to predict rollout behavior and reduce long-tail latency in synchronous LLM RL.
    * RobustRL: Role-Based Fault Tolerance System for RL Post-Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/chen-zhenqian)]
      * ZJU & SKLMEAC
      * Design role-based fault isolation for RL post-training so trainer, rollout, and management failures can recover without restarting the whole job.
  * MoE Training
    * Tessera: A Holistic Pipeline Parallelism Framework for Trillion-Parameter Heterogeneous MoE Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/hu-weifang)]
      * HUST & Alibaba Cloud
      * **Operational Systems**
      * Present **Tessera**, a pipeline-parallel training framework for heterogeneous trillion-parameter MoE models with communication-aware partitioning and runtime rebalancing.
  * Parallelism
    * Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations \[[Paper](https://www.usenix.org/conference/osdi26/presentation/li-haoyang)]
      * PKU & SJTU & Purdue
      * Extend SPMD training with hierarchical and heterogeneous annotations to handle uneven devices, failures, and data-dependent workload variation.
    * Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap \[[Paper](https://www.usenix.org/conference/osdi26/presentation/qiang)]
      * UCSD & GMU & OpenAI & Meta
      * Introduce **Syncopate**, a compiler/runtime approach that overlaps communication with a fused compute kernel using chunk-centric scheduling.
  * Data Pipeline
    * Teaching the Old Dog New Tricks: Building Efficient Data Pipelines for Large-Scale LLM Pre-Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/chen-luofan)]
      * USTC & ByteDance Seed & THU & IA, HFNL
      * **Operational Systems**
      * Characterize production pre-training data paths and optimize checkpoint replication, hot-file replication, and storage-side data transformation.
  * Reliability and Fault Tolerance
    * SDCs in the Wild: Characterizing and Diagnosing SDC-Defective GPUs in Production LLM Training \[[Personal Notes](/reading-notes/conference/osdi-2026/sdchunter)] \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zheng)]
      * SJTU & ByteDance Seed
      * **Operational Systems**
      * Characterize production SDC-defective GPUs and show why synthetic microbenchmarks miss data-dependent, aging-related, and model-sensitive corruption.
      * Introduce **SDCHunter**, replaying the exact failure-triggering training workload and input data to diagnose defective GPUs.
      * Deployed at ByteDance to mitigate 40 SDC incidents with less than 4% training overhead, 100% coverage and device-level localization for real SDC-defective accelerators, and diagnosis shortened to within one hour.
    * Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours \[[Personal Notes](/reading-notes/conference/osdi-2026/aegis)] \[[Paper](https://www.usenix.org/conference/osdi26/presentation/lei)]
      * THU & ByteDance
      * Present **AEGIS**, an online SDC detection framework that separates lightweight corruption sensing from definitive verification during LLM training.
    * OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zhou-ziming)]
      * UMich & ByteDance Seed
      * Introduce **OpGuard**, using bitwise alignment across runs as a production debugging primitive for locating silent LLM training divergence.
    * TrainMover: An Interruption-Resilient Runtime for ML Training \[[Paper](https://www.usenix.org/conference/osdi26/presentation/lao)]
      * Harvard & Alibaba & UT Austin
      * Present **TrainMover**, using elastic and standby machines to recover large-scale training jobs from interruptions with low downtime and no memory overhead.
* LLM Inference
  * Long-Context Inference
    * Strata: Hierarchical Context Caching for Long Context Language Model Serving \[[Paper](https://www.usenix.org/conference/osdi26/presentation/xie-zhiqiang)]
      * Stanford & SJTU & CU Boulder & CMU & NVIDIA & UMich
      * Introduce **Strata**, a hierarchical context caching framework with GPU-assisted I/O and cache-aware scheduling for long-context LLM serving.
    * No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs \[[Paper](https://www.usenix.org/conference/osdi26/presentation/luo)]
      * UVA
      * Present **DirectKV**, a zero-copy KV cache offloading design that lets GPU kernels access CPU-resident KV blocks over high-bandwidth CPU-GPU interconnects.
  * KV Cache Management
    * ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs \[[Paper](https://www.usenix.org/conference/osdi26/presentation/liu-guangda)]
      * SJTU & Huawei & Guizhou University
      * Propose **ECHO**, combining graph-friendly KV cache management with lossless prefetching for native sparse-attention LLM serving.
  * Request Scheduling
    * Simple Is Better: Multiplication May Be All You Need for LLM Request Scheduling \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zhang-dingyan)]
      * SJTU & Alibaba
      * Use a tuning-free multiplicative score over KV-cache reuse and instance load to route requests across LLM serving instances.
    * Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning \[[Paper](https://www.usenix.org/conference/osdi26/presentation/yu-shan)] \[[Code](https://github.com/ovg-project/kvcached)]
      * UCLA & UC Berkeley & Harvard & CMU & Edinburgh & Intel & Stanford & LMSYS & ByteDance & Alibaba Cloud & THU & Novita AI & Rice
      * Introduce **Prism**, a memory-centric multi-model serving framework that uses GPU memory ballooning to adapt spatial and temporal sharing across bursty model groups.
  * Disaggregated Serving
    * Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration \[[Paper](https://www.usenix.org/conference/osdi26/presentation/du)]
      * SYSU
      * Present **EcoServe**, a partially disaggregated serving strategy that coordinates commodity GPU instances to reduce prefill-decode interference without relying on high-end interconnects.
    * Revisiting Pipeline Parallelism for LLM Serving \[[Paper](https://www.usenix.org/conference/osdi26/presentation/hwang)]
      * Korea University
      * Revisit pipeline parallelism for online LLM serving with dynamic chunk sizing and delay scheduling to reduce pipeline bubbles.
    * OpenTela: Unifying Decentralized Computing Resources for Heterogeneous LLM Serving \[[Paper](https://www.usenix.org/conference/osdi26/presentation/yao)]
      * ETH & Cambridge & EPFL & MIT & HKUST
      * **Operational Systems**
      * Present **OpenTela**, a user-space orchestration overlay for turning fragmented HPC clusters into a fault-tolerant heterogeneous LLM serving platform.
    * Kairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing \[[Paper](https://www.usenix.org/conference/osdi26/presentation/jiang-yapeng)]
      * SYSU & Peng Cheng Laboratory & EPFL & Qilu University of Technology
      * Propose **Kairox**, an adaptive GPU-CPU inference system that redistributes FFN neurons online based on activation patterns and runtime bottlenecks.
  * MoE Inference
    * Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU–GPU Hybrid Design \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wang-wenxin)]
      * THU & Xingyun
      * Design a CPU-GPU hybrid MoE inference stack with stream-loading prefill, local prefill-decode disaggregation, and CPU-optimized FP8 execution.
    * UEP: Portable Expert-Parallel Communication \[[Paper](https://www.usenix.org/conference/osdi26/presentation/mao-ziming-uep)]
      * UC Berkeley & UC Davis & UW-Madison & AMD & THU & AWS & ICSI & Broadcom & UPB
      * Present **UEP**, a portable expert-parallel communication system that replaces GPU-initiated RDMA with a GPU-CPU control channel and CPU proxy transport.
    * BatchGen: An Architecture for Scalable and Efficient Batch Inference \[[Paper](https://www.usenix.org/conference/osdi26/presentation/xu-tairan)]
      * Edinburgh & Tencent
      * Introduce a sequence-coroutine compute model for large-scale batch inference, enabling dynamic work reorganization across GPUs and memory-constrained accelerators.
    * UCCL-Tran: An Extensible Software Transport Layer for GPU Networking \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zhou-yang)]
      * UC Berkeley & UC Davis & THU & Harvard & IBM Research & AWS & Broadcom & UPB
      * Propose **UCCL-Tran**, a software transport layer that moves GPU networking control paths to host CPUs for extensible multipath and collective communication policies.
  * Quantization
    * ADAngel: Accelerating Arbitrary-Precision Quantized LLMs with Adaptive Computing Mapping \[[Paper](https://www.usenix.org/conference/osdi26/presentation/liu-yao)]
      * SJTU
      * Build **ADAngel**, a workload-adaptive mixed-precision GEMM engine that selects specialized kernels for arbitrary-precision quantized LLM inference.
  * Tracing and Diagnosis
    * StriaTrace: Efficient Tracing and Diagnosis for Online LLM Inference \[[Paper](https://www.usenix.org/conference/osdi26/presentation/wu-haonan)]
      * SJTU & Alibaba
      * **Operational Systems**
      * Introduce **StriaTrace**, a low-overhead tracing and diagnosis system for online LLM inference anomalies using critical-path tracing and regression-based roofline analysis.
  * Tree-of-Thought Reasoning
    * Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zhong)]
      * PKU & ByteDance Seed
      * Propose **SPEX**, speculatively exploring Tree-of-Thought reasoning paths to break reward-synchronization bottlenecks.
  * Agentic Workflow Serving
    * Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms \[[Paper](https://www.usenix.org/conference/osdi26/presentation/chaudhry)]
      * MIT & Microsoft Azure
      * Present **Murakkab**, an orchestration system that exposes agentic workflows for holistic accuracy, latency, energy, and cost optimization.
* LLM Operations
  * Code Optimization
    * ECO: An AI-Driven Code Efficiency Optimizer for Warehouse Scale Computers \[[Paper](https://www.usenix.org/conference/osdi26/presentation/lin-hannah)]
      * Google DeepMind & Google
      * **Operational Systems**
      * Build **ECO**, a production code-efficiency optimization workflow that localizes opportunities and controls LLM-generated change risk.
  * Performance Diagnosis
    * Diagnosing Performance Issues in Application-Defined Resources \[[Paper](https://www.usenix.org/conference/osdi26/presentation/hu-yigong)]
      * BU & UW & EPFL & UCLA
      * Present **gigiprofiler**, combining LLM-based semantic inference with static and runtime analysis to diagnose bottlenecks in application-defined resources.

### Deep Learning Compilation

* GPU Kernels and Compilers
  * Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs \[[Paper](https://www.usenix.org/conference/osdi26/presentation/soi)]
    * Stanford & NVIDIA
    * Analyze how software pipelining and warp specialization interact on tensor-core GPUs and derive optimal scheduling strategies.
  * TileLoom: Automatic Dataflow Planning for Tile-Based Languages on Spatial Dataflow Accelerators \[[Paper](https://www.usenix.org/conference/osdi26/presentation/li-wei)]
    * NUS & ASU & Google & Lumai
    * Introduce **TileLoom**, an automatic dataflow planning system for tile-based languages on spatial dataflow accelerators.
  * MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs \[[Paper](https://www.usenix.org/conference/osdi26/presentation/cheng)]
    * CMU & THU & NVIDIA & UMich & PKU
    * Present **MPK**, compiling multi-GPU inference into persistent mega-kernels with SM-level task graphs and cross-operator pipelining.
  * GraCE: Unlocking CUDA Graphs with Compiler Support for ML Workloads \[[Paper](https://www.usenix.org/conference/osdi26/presentation/ghosh)]
    * IISc & MSR
    * Propose **GraCE**, a compiler framework that expands CUDA Graph coverage for ML workloads through automatic code transformations and graph-safe execution.
  * VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination \[[Paper](https://www.usenix.org/conference/osdi26/presentation/hu-muyan)]
    * UIUC & Georgia Tech & Microsoft
    * Introduce virtual tensors as a DNN compilation abstraction for eliminating unnecessary data movement across tensor operators.

### Resource Management

* Cluster Scheduling
  * Heterogeneity at Hyperscale: Characterization and Scheduling of Large Production AI Clusters at Alibaba \[[Paper](https://www.usenix.org/conference/osdi26/presentation/li-suyi)] \[[Trace](https://github.com/alibaba/clusterdata/tree/master/cluster-trace-gpu-v2026)]
    * HKUST & Alibaba & Fudan
    * **Our work!**
    * **Operational Systems**
    * Characterize a six-month production AI-cluster trace and design scheduling mechanisms for stranded capacity, locality constraints, and heterogeneous GPUs.
  * DVLA: Dynamic VM Lifetime Aware Scheduling for Drifting Lifetime Distributions and Long-Lived VM Placement Debt \[[Paper](https://www.usenix.org/conference/osdi26/presentation/zhang-zhengtong)]
    * Alibaba Cloud
    * **Operational Systems**
  * PIMS: Fleet-Wide Datacenter Maintenance with Minimal Capacity Buffer and Predictable Latency \[[Paper](https://www.usenix.org/conference/osdi26/presentation/leonhardi)]
    * Meta & Cambridge & OSU & NexGeMM
    * **Operational Systems**
  * Mimesys: Generating Realistic Executable Testing Environments from Resource Usage Traces \[[Paper](https://www.usenix.org/conference/osdi26/presentation/kim-donghyun)]
    * UT Austin
* Resource Allocation
  * Quota Marketplace: Dynamic Pricing for Efficient Allocation of ML Training Resources \[[Paper](https://www.usenix.org/conference/osdi26/presentation/sivan)]
    * Google & Drexel
    * Present a deployed market-based mechanism for dynamically pricing and allocating scarce ML training chips under heterogeneous workload value.

### Serverless Computing

* Stop Pretending to Be Busy: A Case for Serverless Paradigms in Co-Located Batch Workloads \[[Paper](https://www.usenix.org/conference/osdi26/presentation/chai)]
  * THU & Ant Group & SJTU & Quan Cheng Laboratory
  * **Operational Systems**
  * Present **Quark**, applying serverless-style fine-grained allocation, skew-aware scheduling, and rapid provisioning to co-located batch analytics.
* Continuation-Centric Computing with Arca \[[Paper](https://www.usenix.org/conference/osdi26/presentation/srivatsan)]
  * Stanford
  * Introduce **Arca**, an OS design that captures portable continuations so short-lived functions can be paused, migrated, copied, and resumed.
* Rethinking Process Snapshots for Near-Warm Serverless Cold Starts \[[Paper](https://www.usenix.org/conference/osdi26/presentation/holmes)]
  * MIT & UPenn
  * Build **Spice**, combining a snapshot file format with kernel restore primitives to reduce serverless cold-start overhead from disk snapshots.

## Acronyms

* KV: Key-Value
* LLM: Large Language Model
* MoE: Mixture-of-Experts
* RL: Reinforcement Learning
* SDC: Silent Data Corruption
* SLO: Service Level Objective
* SPMD: Single Program Multiple Data


# SDCs in the Wild: Characterizing and Diagnosing SDC-defective GPUs in Production LLM Training

## Meta Info

Presented in [OSDI 2026](https://www.usenix.org/conference/osdi26/presentation/zheng).

Authors: Wenxin Zheng, Wenxiao Wang, Yun Zhang, Mingcong Han, Bin Xu, Jinyu Gu, Xingda Wei, Haibo Chen, Zuquan Song, Gaohong Liu, Yucheng Nie, Zhe Nan, Zhuolin Zheng, Huan Yu, Shuguang Wang, Ziming Zhou, Hang Zhu, Wencong Xiao, Xin Liu (*SJTU, ByteDance Seed*)

## Understanding the paper

### TL;DR

**SDCHunter** is a production diagnosis system for SDC-defective GPUs in large-scale LLM training. The paper first characterizes 23 real defective GPUs from ByteDance production clusters and shows why generic synthetic stress tests miss many faults: GPU SDCs often emerge after deployment due to aging, depend on exact kernels/input values/precision modes, and remain invisible to ECC, thermal sensors, and common hardware counters.

The key design is exact deterministic replay of the workload that triggered the anomaly. SDCHunter first uses lightweight cross-replica signatures at pipeline-parallel communication boundaries to isolate a suspicious group, then performs fine-grained replay comparison inside that group to locate the defective GPU.

### Background

* Silent Data Corruption (SDC) silently produces wrong values without an explicit hardware or software alarm.
* In production LLM training, SDC can appear as:
  * fail-stop errors, such as shape mismatch or out-of-bound access, where the crash site is downstream from the true corrupted kernel;
  * implicit training degradation, such as loss spikes that may pollute later checkpoints.
* The same defective GPU can have very different manifestation rates. The paper reports replay occurrence rates ranging from 100% down to 1e-6%, explaining why fixed-duration stress tests are unreliable.

### Key observations

* **Lifecycle:** SDC-defective GPUs are not only caught during burn-in. In the 23-GPU study, 25% were detected during pre-deployment burn-in, 25% within the first two months, 10% within six months, and 40% around one year after deployment.
* **Microarchitecture and operator affinity:** First visible corruptions appear in kernels such as GEMM, InplaceMoE, FlashAttention, RMSNorm, Scatter, and fused kernels. For GEMM cases, FP32/FP64 CUDA cores showed higher vulnerability than some lower-precision Tensor Core paths.
* **Input sensitivity:** The same operator on the same defective GPU may fail only for particular data ranges, data types, or value patterns.
* **Low observability:** The corruptions can be subtle value biases, and standard signals such as ECC counters, machine-check exceptions, performance counters, clock frequency, and thermal limits do not distinguish defective GPUs from healthy ones.

### Challenges

* **Replay determinism:** Differential testing only works if healthy runs are bitwise identical. LLM training breaks this through nondeterministic GPU operators, communication topology changes, NCCL reduction-order differences, and scale-dependent kernel/parallelism choices.
* **Observability vs. overhead:** Loss-level metrics can mask early corruption, while kernel-level instrumentation can change timing enough to hide faults. The paper reports kernel-granularity inspection with 683.9% overhead, compared with 4.3% overhead for communication-boundary checks but much lower diagnostic resolution.
* **Scale:** Replaying a full production job on the same number of GPUs is impractical, but scaling down can change the training behavior enough to obscure whether a discrepancy is from SDC or from the changed configuration.

### Design

* **Deterministic training prerequisite**
  * Fix random seeds and deterministic framework toggles.
  * Disable autotuning and kernel-selection heuristics that change execution plans.
  * Serialize or reimplement nondeterministic operators when needed.
  * Lock communication algorithms, channel counts, and reduction order.
  * Record and replay the same communication schedule used by the original run.
  * In ByteDance production measurements, deterministic training had less than 0.01% step-time difference and reduced normalized debugging time from 1.00x to 0.30x.
* **Phase 1: lightweight grouping**
  * Partition the cluster along the data-parallel dimension into replay replicas that consume the same input batch.
  * Hash tensor values at framework-visible pipeline-parallel communication boundaries.
  * Compare compact signatures across replicas to find the divergent DP group.
  * Remove the suspicious group from the training path and resume from the latest valid checkpoint before device-level localization finishes.
* **Phase 2: precise localization**
  * Replay the problematic iteration on the suspicious group and a healthy reference group.
  * Collect layer-wise signatures for intermediate tensors.
  * Find the first divergent tensor/kernel and map it to the owning rank and physical GPU.
  * Confirm suspicious devices offline through iterative replay of the alert-triggering trace and hardware tools.

### Evaluation

* SDCHunter is deployed as ByteDance production training infrastructure and had identified 40 SDC-defective GPUs by paper submission.
* Production diagnosis time is reduced from several days to under one hour; offline device confirmation also finishes within one hour and does not block the resumed training job.
* End-to-end overhead stays below 4% for 50B and 150B model traces on 128/512 GPUs.
* In real-world trace evaluation, SDCHunter covers all confirmed SDC-defective GPUs; in fault injection, it reaches 91% coverage and detects faults within 3 steps on average.
* Device-level localization accuracy is 100% in both real-world and fault-injection evaluations.

### Takeaways

* Exact production-workload replay is more useful than generic microbenchmarks for data-dependent GPU SDCs.
* SDC diagnosis should be integrated with the training control plane: isolate suspicious machines quickly, resume training, and defer expensive device-level confirmation.
* Deterministic training is not only a reliability mechanism for SDCs; it also helps separate model, data, infrastructure, operator, and hardware causes during production debugging.


# Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours

## Meta Info

Presented in [OSDI 2026](https://www.usenix.org/conference/osdi26/presentation/lei).

Authors: Kinman Lei, Liyan Zheng, Xiang Li, Hongmin Chen, Yun Zhang, Gaohong Liu, Zuquan Song, Zixuan Ma, Zhiyu Xue, Minghui Yu, Shuguang Wang, Wencong Xiao, Haibin Lin, Yuyang Jin, Jidong Zhai, Bo Liu, Xin Liu (*THU, ByteDance*)

## Understanding the paper

### TL;DR

**AEGIS** is an online SDC detection framework for production LLM training. Instead of stopping the job for offline diagnostics or fully duplicating computation, it splits detection into two stages: a lightweight inline **cSensor** that captures suspicious evidence on the training critical path, and an off-path **cVerifier** that confirms the corruption later.

The system combines two families of sensors: mixed-precision-aware algorithmic checksums for Matmul and FlashAttention, and deterministic self-equivalence checks that reuse recomputation already present in LLM training. In a 3.5e7 GPU-hour deployment, AEGIS detected 18 real SDC incidents involving 13 faulty GPUs with 0.86% production overhead.

### Background

* SDCs in large-scale LLM training are rare per device but frequent at fleet scale.
* Offline diagnostics interrupt training and cover only limited synthetic workloads.
* Rerun-based approaches can confirm corruption but are expensive at 10K-GPU scale.
* Classical algorithm-based online detection struggles with low-precision LLM training because bf16/fp8 round-off noise can dominate checksum differences.

### Observations

* **Non-deterministic reproducibility:** Running the same workload on the same faulty GPU does not always trigger SDC. In one Matmul case, 9.2e5 repeated executions produced only three distinct erroneous outcomes.
* **Workload sensitivity:** SDCs can appear only in specific numerical regimes. Scaling the input tensors of the same Matmul changed whether the fault manifested.
* **Diverse fault patterns:** Faults surface in different operators and expose different evidence. No single checker covered all observed faulty machines.

### Design

* **cSensor-cVerifier abstraction**
  * cSensor runs inline after selected computations, records compact evidence, and emits verification tasks only for suspicious computations.
  * vTask is the shared interface between heterogeneous sensors and the verifier. It stores the minimal replay/fingerprint context plus metadata such as operator names and indices.
  * cVerifier consumes vTasks off the critical path, using pipeline bubbles and a small reserved time slice at step boundaries to avoid adding synchronization stalls.
  * AEGIS reports an SDC only after cVerifier confirms it, reducing false positives from benign numerical noise.
* **Mixed-precision-aware algorithmic detection**
  * For Matmul, AEGIS compares equivalent checksum paths such as `C1 = (AB)1` and `A(B1)`.
  * Instead of accumulating checksums from final bf16 outputs, it accumulates from fp32 Tensor Core accumulators, preserving precision and separating real corruption from normal round-off error.
  * For FlashAttention, AEGIS uses the invariant `1^T dV = 1^T dO`, derived from `dV = P^T dO` and the fact that each softmax row of `P` sums to 1.
* **Self-equivalence-based deterministic detection**
  * LLM training already recomputes some values, for example through activation recomputation and FlashAttention backward recomputation.
  * Under deterministic execution, original and recomputed outputs should be bitwise identical.
  * AEGIS records xorsum fingerprints for the original and recomputed outputs, then lets cVerifier compare the fingerprints.
  * Selective instrumentation checks the final operator in a deterministic recomputation chain, reducing vTask volume while preserving coverage of the chain.
* **Sensing control and deployment knobs**
  * Adaptive thresholds track runtime checksum-difference distributions, raising thresholds after suspicious spikes and decaying them later.
  * Dynamic sampling controls algorithmic-detection overhead by sampling operators with a tunable probability.
  * Outlier warning compares checksum differences across data-parallel ranks and flags extreme deviations as heuristic warnings, separate from confirmed SDC reports.

### Implementation

* Integrated into ByteDance's in-house Megatron-LM training stack with minimal pipeline changes.
* vTask packing keeps only the data needed for later verification:
  * row checksums for forward and input-gradient Matmul;
  * row-and-column checksums for weight-gradient Matmul;
  * compact fingerprints for deterministic recomputation checks.

### Evaluation

* Production deployment covered 3.5e7 GPU-hours and detected 18 SDC incidents on 13 faulty GPUs with 0.86% overhead.
* Only three detected incidents showed observable training failures such as NaN loss; the remaining incidents were silent.
* Deterministic detection caught 12 incidents, while algorithmic detection and outlier warnings caught 6, showing that the sensors are complementary.
* On eight real faulty machines, AEGIS detected all 8, while vendor offline diagnostics detected only 2.
* Runtime overhead averages below 1.61% with row checksums and below 2.35% with row-and-column checksums; maximum slowdown is 2.1% and 2.6%, respectively.
* The system scales to 1,024 GPUs because it avoids extra global communication.
* The fp32-accumulator checksum ablation shows near-100% detection at the median reference injection scale, while bf16-output checksums require roughly 1e4x larger perturbations for comparable sensitivity.

### Discussion

* All production SDC incidents detected by AEGIS were reproducible on the corresponding GPUs, indicating permanent hardware faults rather than purely transient events.
* Rebooting can temporarily hide symptoms but does not fix the underlying fault.
* AEGIS can also catch SDC-like software bugs when they cause deterministic recomputation mismatches.
* Current coverage focuses on dominant Transformer compute kernels such as Matmul and FlashAttention; SDCs outside the protected backbone remain possible.
* AEGIS complements replay-based diagnosis systems such as SDCHunter: AEGIS detects suspicious corruption online, while replay-based tools can perform deeper fault localization and hardware confirmation.


# ICML 2026

## Meta Info

Homepage: <https://icml.cc/Conferences/2026>

Paper list: <https://icml.cc/virtual/2026/papers.html?filter=titles>

## Papers

### Large Language Models (LLMs)

* LLM Inference
  * Agentic Inference
    * ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System \[[Paper](https://openreview.net/forum?id=kR4iOTaAOJ)] \[[arXiv](https://arxiv.org/abs/2602.13692)] \[[Code](https://github.com/ThunderAgent-org/ThunderAgent)] \[[Homepage](https://thunderagent.ai)] \[[ICML](https://icml.cc/virtual/2026/poster/62040)]
      * GaTech & UIUC & CMU & Together AI
      * **Spotlight**
      * Introduce **ThunderAgent**, an agentic inference and rollout system that represents workflows as LLM Programs with unified KV-cache, state, and tool-resource metadata.
      * Use program-aware scheduling and tool resource management to improve KV-cache reuse, reduce memory imbalance, and prepare tool environments asynchronously.
  * Workload Characterization
    * Characterizing Agents in Production \[[Paper](https://openreview.net/forum?id=mWxEAgz3xu)] \[[arXiv](https://arxiv.org/abs/2512.04123)] \[[ICML](https://icml.cc/virtual/2026/oral/71172)]
      * UC Berkeley & Intesa Sanpaolo & UIUC & Stanford & IBM Research
      * **Oral**
      * Present **CAP**, a systematic study of production LLM agents using 20 interviews and a survey of 306 practitioners across 26 domains.
      * Find that production agents often use simple, controllable approaches: bounded step counts before human intervention, off-the-shelf prompting, and human evaluation.


# ISCA 2026

## Meta Info

Homepage: <https://iscaconf.org/isca2026/>

## Papers

### Kernel Generation

* KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta \[[Personal Notes](/reading-notes/conference/isca-2026/kernelevolve)] \[[arXiv](https://arxiv.org/abs/2512.23236)] \[[Blog](https://engineering.fb.com/2026/04/02/developer-tools/kernelevolve-how-metas-ranking-engineer-agent-optimizes-ai-infrastructure/)]
  * Meta
  * Present **KernelEvolve**, an agentic kernel coding framework that automates kernel generation and optimization from kernel specifications across heterogeneous AI accelerators.
  * Combine tree-search-based kernel exploration, retrieval-augmented hardware knowledge injection, and profiling-driven evaluation feedback in a single optimization loop.
  * Validate 100% correctness on 160 ATen operators across H100, MI350, and MTIA v3, and achieve a 100% pass rate on all 250 KernelBench problems.


# KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta

\#kernel\_generation #heterogeneous\_accelerators #dlrm

## Meta Info

Presented in [arXiv:2512.23236](https://arxiv.org/abs/2512.23236).

Authors: KernelEvolve Team (*Meta*)

Resources: [Blog](https://engineering.fb.com/2026/04/02/developer-tools/kernelevolve-how-metas-ranking-engineer-agent-optimizes-ai-infrastructure/)

## Understanding the paper

### TL;DR

* KernelEvolve is an agentic kernel coding system for recommendation-model training and inference across NVIDIA GPUs, AMD GPUs, Meta MTIA chips, and CPUs.
* The core idea is to formulate kernel generation as a graph/tree search problem instead of one-shot code generation.
  * The system iteratively generates candidate kernels, evaluates them, retrieves hardware-specific knowledge, and feeds diagnostics back into the next round of synthesis.
* It closes the loop with a strong tooling stack.
  * Correctness and speed are validated automatically with TritonBench and cross-stack profilers such as Torch Profiler, NCU, Proton, and MTIA Insight.
* The reported outcomes are strong.
  * 100% pass rate on all 250 KernelBench tasks.
  * 100% correctness on 160 ATen operators across three hardware platforms.
  * Up to 17x speedup over PyTorch baselines on production kernels, while shrinking optimization time from weeks to hours.

### Background

* The paper focuses on Meta's DLRM and ranking workloads.
  * These workloads span multiple model generations, from embedding-heavy recommenders to sequence-learning models and more recent large-scale ranking models.
* Kernel optimization has a three-dimensional explosion.
  * Hardware diversity: NVIDIA GPUs, AMD GPUs, MTIA, and multiple hardware generations.
  * Model diversity: different recommendation architectures need different operators.
  * Kernel diversity: many production operators are custom preprocessing, fusion, or ranking-specific kernels outside vendor libraries.
* This makes manual kernel tuning a bottleneck.
  * A kernel optimized for one hardware generation may not transfer well to another.
  * Vendor libraries and compiler autotuning can cover common kernels, but they do not fully cover the long tail of production operators.

### Challenges

* Hardware heterogeneity
  * Different accelerators expose different memory hierarchies, instruction sets, execution models, and profiling tools.
  * Proprietary hardware such as MTIA is absent from public LLM training corpora, so a generic coding assistant lacks the required hardware knowledge.
* Model and operator diversity
  * Production recommendation systems use many workload-specific operators, including data preprocessing and fused business logic, which makes the search space much larger than standard GEMM/conv tuning.
* Cross-stack optimization difficulty
  * Useful performance signals are fragmented across Triton/DSL code, compiler IR, runtime traces, and low-level hardware counters.
  * A practical kernel-coding agent needs an automated way to correlate these signals and use them in the next search iteration.

### Existing approaches

* Vendor libraries
  * Libraries such as cuBLAS and cuDNN work well for standard operators, but they do not solve the long tail of custom ranking kernels.
* Compiler autotuning and fusion
  * These approaches help explore scheduling and fusion spaces, but still struggle to cover the full combination of shapes, hardware targets, and custom operators at Meta's scale.
* One-shot LLM code generation
  * A single draft is usually not enough for kernel optimization because correctness bugs, profiling bottlenecks, and hardware-specific constraints must be resolved iteratively.

### Designs

<figure><img src="https://819228986-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkzeiawY8SkBarQBDVm-659326392%2Fuploads%2Fgit-blob-016b9834f3152996c05df990758e49cf51ddab8a%2FKernelEvolve-overview.png?alt=media" alt=""><figcaption><p>KernelEvolve system architecture and optimization workflow.</p></figcaption></figure>

* **Search formulation**
  * KernelEvolve models the optimization process as a search graph.
  * Each node is a kernel artifact, each edge is a transformation, and the system repeatedly applies selection, transformation, and fitness evaluation.
  * The paper mentions multiple search strategies, including greedy search, Monte Carlo Tree Search (MCTS), and evolutionary algorithms.
* **LLM synthesizer with dynamic prompts**
  * The system generates kernels across multiple abstractions, from Triton and CuTe-style DSLs to lower-level backends such as CUDA, HIP, and MTIA C++.
  * Prompts are not static templates. They are enriched online with runtime diagnostics, retrieved hardware knowledge, and prior search history.
* **Agentic retrieval and context management**
  * A deep-search sub-agent retrieves relevant documents, code samples, and optimization guidance from a persistent knowledge base.
  * A context-memory sub-agent decides what historical context from the search tree should be kept for the next iteration.
  * This lets the search inherit useful parent/sibling information while still being able to restart and escape local optima.
* **Knowledge injection for proprietary hardware**
  * The knowledge base stores correctness constraints, platform-agnostic optimization guidance, and hardware-specific documentation.
  * For MTIA, the system injects architecture manuals, instruction references, memory hierarchy information, and optimization patterns at inference time.
  * This is the key mechanism that makes code generation feasible even on hardware unseen during LLM pretraining.
* **Evaluation and debugging loop**
  * TritonBench checks correctness against PyTorch references and measures speedup.
  * Torch Profiler captures system-level timelines.
  * NCU, Proton, and MTIA Insight provide lower-level kernel and hardware-counter views.
  * Meta's MPP (Multi-Pass Profiler) acts as the federated tooling layer that unifies instrumentation, profiling, and trace synthesis across the stack.

### Implementation

* KernelEvolve is organized as a long-running optimization harness rather than an interactive one-shot coding assistant.
* The system stores both metadata and kernel artifacts across optimization runs.
  * Metadata records parent-child relations, quality scores, and whether a candidate is buggy.
  * The object store preserves generated kernels and analysis reports so later runs can reuse prior optimization history.
* The paper also positions the system as self-improving.
  * Successful optimization patterns can be distilled back into the shared knowledge base.
  * Optimization sessions also create structured trajectories that can be reused for post-training smaller domain-specific models.

### Evaluation

* OSS operator coverage
  * KernelEvolve generates Triton kernels for 160 ATen operators on H100, MI350, and MTIA v3.
  * It achieves 100% correctness across all 480 operator-platform combinations.
  * On KernelBench, it reaches a 100% pass rate across all three difficulty levels.
* Search behavior
  * The paper separates the search into a draft phase and a tree-expansion phase.
  * Early iterations sample candidates without memory, while later iterations exploit execution feedback from ancestors to refine promising directions.
* Production case studies
  * Across production workloads, the paper reports 1.2x-17x speedups over PyTorch baselines.
  * For the convolutional-transformer case on H100, KernelEvolve achieves 2.30x over `torch.conv1d` and 1.62x over the optimized `conv2d` workaround on the main FP16 production shape.
  * The main win comes from kernel fusion and eliminating extra layout-conversion kernels rather than only improving raw convolution throughput.
* End-to-end business impact
  * The accompanying Meta blog reports over 60% inference throughput improvement for the Andromeda ads model on NVIDIA GPUs.
  * It also reports over 25% training throughput improvement for an ads model on MTIA.

### Limitations and future work

* The optimized kernels can be highly shape-specialized.
  * In the 1D convolution case study, kernels tuned for production shapes underperform on some out-of-distribution shapes.
* The system still depends on high-quality evaluation infrastructure.
  * Cross-stack profiling, correctness checking, and hardware-specific diagnostics are essential to make the search effective.
* New hardware still requires curated documentation.
  * KernelEvolve reduces the work from hand-writing kernels to curating hardware knowledge, but this knowledge-injection step remains necessary.
* The broader opportunity is larger than kernel coding.
  * The blog and paper suggest extending the same agentic loop to compiler optimization, memory management, and other system-tuning problems.


# CAIS 2026

## Meta Info

Homepage: <https://www.caisconf.org>

Paper list: <https://www.caisconf.org/program/2026/papers/>

Proceedings: <https://dl.acm.org/doi/proceedings/10.1145/3786335>

## Papers

### LLM Inference

* XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813124)]
  * SJTU & CMU
  * Extend XGrammar with dynamic grammar support for efficient structured output generation in agentic LLM workloads (e.g., tool calling with runtime-defined schemas).
* Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813146)]
  * Stanford
  * Replace KV cache with spectral Koopman operator estimation for constant-memory associative recall.
* Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813134)]
  * LinkedIn & MIT
  * Optimize LLM query routing at the batch level under joint cost and capacity constraints for multi-model serving.
* Understanding and Improving Communication Performance in Multi-node LLM Inference \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813165)]
  * UMD & LLNL
  * Characterize and optimize inter-node communication bottlenecks in multi-node LLM inference deployments.

### Diffusion Model Inference

* SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813174)]
  * UofT & Amazon & NVIDIA & AWS
  * Introduce scalable sequence parallelism for distributed inference of Diffusion Transformers (DiTs) across multiple GPUs.

### LLM Optimization

* Scaling Textual Gradients via Sampling-Based Momentum \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813168)]
  * UChicago & UT Austin & Santa Clara & Princeton & MSR & SylphAI
  * Scale textual gradient optimization (TextGrad) via sampling-based momentum for improved convergence.
* optimize\_anything: Unified Text Optimization can Outperform Specialized Systems \[[Paper](https://dl.acm.org/doi/10.1145/3786335.3813167)] \[[Code](https://github.com/gepa-ai/gepa)]
  * UC Berkeley & MIT
  * A unified text optimization framework that subsumes prompt optimization, agent workflow design, and DSPy-style program synthesis into a single search procedure.

## Acronyms

* DiT: Diffusion Transformer
* KV: Key-Value
* LLM: Large Language Model
* LoRA: Low-Rank Adaptation


# MLSys 2026

## Meta Info

Homepage: <https://mlsys.org/Conferences/2026>

Paper list: <https://mlsys.org/virtual/2026/papers.html?filter=titles>

### Acceptance Rate

26.8% (= 135 / 504)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Parallelism
    * AXLearn: Modular, Hardware-Agnostic Large Model Training \[[Paper](https://openreview.net/forum?id=41x11EB3bc)]
      * Google
      * Present **AXLearn**, a production system for scalable large model training with modular software interfaces and hardware-agnostic support across TPUs and GPUs.
    * NEST: Network- and Memory-Aware Device Placement for Distributed Deep Learning \[[Paper](https://openreview.net/forum?id=jpIoO2zSKA)]
      * UW
      * Propose **NEST**, a device placement framework that jointly optimizes parallelism strategy, memory, and network topology for distributed training.
    * veScale-FSDP: Flexible and High-Performance FSDP at Scale \[[Paper](https://openreview.net/forum?id=3Lj8R0F48P)]
      * ByteDance & THU
      * Present **veScale-FSDP**, a flexible FSDP system supporting block-structured sharding formats for structure-aware training methods like block-wise quantization and Shampoo optimizers.
    * DreamDDP: Accelerating Low-Bandwidth Geo-Distributed LLM Training with Layer-wise Partial Synchronization \[[Paper](https://openreview.net/forum?id=cnvw0mbZQp)]
      * HKUST-GZ
      * Propose **DreamDDP** with layer-wise partial synchronization to accelerate geo-distributed LLM training under low-bandwidth conditions beyond Local SGD.
    * Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP \[[Paper](https://openreview.net/forum?id=MPVycRsIn6)]
      * CMU & UC Berkeley
      * Propose **FCP**, a flexible context parallelism system that adaptively handles variable sequence lengths with efficient sharding and bin-packing for pre-training.
    * Efficient Long-Context Language Model Training by Core Attention Disaggregation \[[Paper](https://openreview.net/forum?id=oIonqkc8hM)]
      * UC Berkeley & UCSD
      * Present core attention disaggregation (**CAD**), scheduling the parameter-free softmax(QK^T)V computation on a separate resource pool to resolve load imbalance in long-context training.
    * MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training \[[Paper](https://openreview.net/forum?id=h6SD2zgwGq)]
      * MSR
      * Propose **MTraining**, a distributed dynamic sparse attention system for efficient ultra-long context LLM training in multi-GPU settings.
  * Low-Rank Training
    * BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models \[[Paper](https://openreview.net/forum?id=JhN5hldx4V)]
      * Argonne National Lab
      * Propose **BOOST**, a scalable training framework for low-rank LLMs that addresses excessive communication and poor GPU utilization under standard tensor parallelism.
  * Memory Management
    * ProTrain: Efficient LLM Training via Automatic Memory Management \[[Paper](https://openreview.net/forum?id=XDkOn0iTiH)]
      * UMass Amherst
      * Introduce **ProTrain**, an automatic memory management system that adaptively configures memory-saving techniques for LLM training without manual tuning or system expertise.
  * Fault Tolerance
    * GUARD: Scalable Straggler Detection and Node Health Management for Large-Scale Training \[[Paper](https://openreview.net/forum?id=JFEwQ821MS)]
      * Meta
      * Present **Guard**, a scalable system for detecting fail-slow stragglers and managing node health during frontier-scale foundation model training on tens of thousands of GPUs.
    * Sparing Strategies to Minimize Reliability Impact On Large Training Jobs \[[Paper](https://openreview.net/forum?id=18jPgte2tM)]
      * Meta
      * Present an analytical framework for optimizing sparing strategies (compute block size, spare count, spare GPU trays) to minimize reliability impact on Meta's large-scale LLM training jobs.
  * RL Post-Training
    * Beat the long tail: Distribution-Aware Speculative Decoding for RL Training \[[Paper](https://openreview.net/forum?id=kMeqqPBjSl)]
      * Together AI & Stanford & UCSD
      * Identify rollout length long-tail as a major bottleneck in RL training and apply distribution-aware speculative decoding to accelerate the generation phase.
    * HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments \[[Paper](https://openreview.net/forum?id=LRLyuaz1W7)]
      * Amazon
      * Propose **HetRL**, a system for efficient LLM reinforcement learning across heterogeneous GPU environments with multi-generation hardware.
    * ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems \[[Paper](https://openreview.net/forum?id=HhDSxs7x2R)]
      * NTU & HPC-AI Tech
      * Identify critical gaps in integrating speculative decoding into RL systems (diminishing speedups, drafter staleness, draft-verification misalignment) and propose **ReSpec** to address them.
  * Heterogeneous Training
    * HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware \[[Paper](https://openreview.net/forum?id=KgcqSNio0U)]
      * PKU & HKUST & Cambridge
      * Propose **HexiScale**, a system supporting asymmetric partition of training computations across heterogeneous GPUs with flexible data, tensor, and pipeline parallelism.
    * Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters \[[Paper](https://openreview.net/forum?id=40leuGH3iO)]
      * U Waterloo & Microsoft
      * Propose **Zorse** for optimizing LLM training on heterogeneous GPU clusters through careful workload partitioning across different GPU generations.
    * FlexTrain: Scalable Hybrid-Parallel Training with Elastic Resource Utilization and Consistent Accuracy \[[Paper](https://openreview.net/forum?id=h2yhNcbwSL)]
      * CUHK-SZ & ByteDance
      * Propose **FlexTrain**, an elastic training system that dynamically adjusts GPU allocations to harness idle resources while maintaining accuracy consistency with minimal profiling overhead.
  * Data Pipeline
    * Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework \[[Paper](https://openreview.net/forum?id=ok96wGyPdI)]
      * Meta
      * Present **Matrix**, a peer-to-peer multi-agent framework for scalable synthetic data generation with decentralized orchestration eliminating central bottlenecks.
* LLM Inference
  * Speculative Decoding
    * Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding \[[Paper](https://openreview.net/forum?id=yeqrwcWjPu)]
      * CMU & MIT & UC Berkeley
      * Propose sparse self-speculative decoding that exploits activation sparsity in reasoning models for lossless inference acceleration without auxiliary draft models.
    * PRISM: Parametrically Refactor Inference for Speculative Decoding Draft Models \[[Paper](https://openreview.net/forum?id=cvU2HuuxEf)]
      * CUHK & ICT, CAS
      * Propose **PRISM**, which parametrically refactors draft models for speculative decoding to balance draft quality and computational overhead.
    * SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding \[[Paper](https://openreview.net/forum?id=o42VU86ZsV)]
      * UVA
      * Propose **SpecDiff-2**, scaling diffusion drafter alignment for speculative decoding to enable parallel drafting and reduce token rejections.
    * Speculative Decoding: Performance or Illusion? \[[Paper](https://openreview.net/forum?id=fzkqtezFEi)]
      * UC Berkeley
      * Present the first systematic study of speculative decoding on production-grade inference engines (vLLM), revealing practical limitations at realistic batch sizes and workloads.
  * KV Cache Management
    * Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost \[[Paper](https://openreview.net/forum?id=r3mQiuYKIN)]
      * Together AI & U Sydney
      * Propose **Kitty**, an algorithm-system co-design for mixed-precision 2-bit KV cache quantization with dynamic channel-wise precision boost that maintains accuracy on long-context reasoning.
    * FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management \[[Paper](https://openreview.net/forum?id=GgX6dPJx9M)]
      * UC Irvine
      * Propose **FlexiCache**, exploiting temporal stability variations of critical tokens across KV heads for efficient per-head cache management policies.
    * OPKV: A High-Throughput Plugin-Driven Framework for Recallable Sparsity in Paged KV Cache Systems \[[Paper](https://openreview.net/forum?id=EB5bgzv4qA)]
      * SEU & GDUT
      * Propose **OPKV**, a high-throughput plugin-driven framework for recallable KV cache sparsity that scales recall efficiency under high batch sizes.
    * SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models \[[Paper](https://openreview.net/forum?id=cJcZKzdwkP)]
      * USC & Intel
      * Propose **SkipKV**, selectively skipping KV generation and storage for thinking tokens in reasoning models to reduce cache overhead without degrading accuracy.
    * Using Span Queries to Optimize Cache and Attention Locality \[[Paper](https://openreview.net/forum?id=qcGGSXpFcM)]
      * IBM Research
      * Propose span queries as a unified abstraction for non-linear KV cache access patterns, improving cache hit rates across diverse inference workflows beyond chat completion.
    * MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing \[[Paper](https://openreview.net/forum?id=PDu13oOl4G)]
      * GMU & UVA
      * Present **MorphServe**, a dynamic LLM serving framework that swaps quantized layers and resizes KV caches at runtime to adapt to workload fluctuations while meeting SLOs.
  * Request Scheduling
    * PLA-Serve: A Prefill-Length-Aware LLM Serving System \[[Paper](https://openreview.net/forum?id=dzjCkSEDyG)]
      * MBZUAI
      * Propose **PLA-Serve**, a prefill-length-aware serving system that disaggregates requests by prompt length for adaptive scheduling to reduce TTFT latency.
    * SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips \[[Paper](https://openreview.net/forum?id=RuslSHdIHa)]
      * Microsoft
      * Propose **SuperInfer**, an SLO-aware serving system with rotary scheduling and NVLink-based KV cache offloading on superchips to sustain responsiveness under high request rates.
    * BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching \[[Paper](https://openreview.net/forum?id=IuVHde07l6)]
      * Microsoft
      * Propose **BatchLLM**, optimizing large-batch offline LLM inference with global prefix sharing and throughput-oriented token batching.
    * Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token \[[Paper](https://openreview.net/forum?id=FuRo7Ur5Ib)]
      * UW
      * Propose **Stream2LLM**, overlapping context retrieval with LLM prefill computation via streaming to reduce time-to-first-token under concurrent requests.
    * BEAM: Joint Resource–Power Optimization for Energy-Efficient LLM Inference under SLO contraints \[[Paper](https://openreview.net/forum?id=BfNBXM8CCT)]
      * KAIST
      * Propose **BEAM**, jointly optimizing resource allocation and power management for energy-efficient LLM inference while meeting latency SLOs.
  * Long-Context Inference
    * ContextPilot: Fast Long-Context Inference via Context Reuse \[[Paper](https://openreview.net/forum?id=RnKvDy1jv2)]
      * U Edinburgh
      * Propose **ContextPilot**, accelerating long-context inference through context reuse that preserves reasoning quality while improving KV cache utilization.
    * MAC-Attention: a Match--Amend--Complete scheme for fast and accurate attention computation \[[Paper](https://openreview.net/forum?id=b6HBRCejb7)]
      * Ohio State & Google
      * Propose **MAC-Attention**, reusing prior attention computations for semantically similar queries to accelerate long-context decoding without fidelity or access loss.
  * Sparsity
    * Attribution-based Sparse Activation in Large Language Models \[[Paper](https://openreview.net/forum?id=gJFigZeb5D)]
      * U Pittsburgh & NTU
      * Propose attribution-based sparse activation for runtime-adaptive lossless neuron deactivation without model retraining, adapting to different tasks and inputs.
    * BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding \[[Paper](https://openreview.net/forum?id=6INSBXTQ4x)]
      * NVIDIA & MIT
      * Introduce **BLASST**, a drop-in dynamic sparse attention mechanism that uses a fixed scalar threshold to skip attention blocks, accelerating long-context inference.
  * Quantization
    * MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design \[[Paper](https://openreview.net/forum?id=VBbMRQ4VOc)]
      * Microsoft
      * Propose **MixLLM**, exploring global mixed-precision quantization between output features based on per-feature importance for efficient LLM compression.
    * Search Your Block Floating Point Scales! \[[Paper](https://openreview.net/forum?id=innqECyZPK)]
      * Cornell & Together AI
      * Propose **ScaleSearch**, optimizing block floating point scales beyond the default max-magnitude choice to reduce quantization error for low-precision inference.
    * CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training \[[Paper](https://openreview.net/forum?id=Fubm1TtWeo)]
      * IST Austria
      * Introduce **CAGE**, a curvature-aware gradient estimation method for quantization-aware training that bridges the accuracy gap with native-precision training via multi-objective optimization.
  * Prefill-Decode Disaggregation
    * Beyond the Buzz: A Pragmatic Take on Inference Disaggregation \[[Paper](https://openreview.net/forum?id=NqC5tcBsa0)]
      * NVIDIA
      * Present the first systematic study of prefill-decode disaggregated inference at scale, revealing the complexity of the optimization space and coordination challenges for production deployment.
    * TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference \[[Paper](https://openreview.net/forum?id=rh2Ylffkq6)]
      * MSR India
      * Propose **TokenWeave**, achieving practical compute-communication overlap for tensor-parallel LLM serving with minimal integration effort into existing engines.
    * TriInfer: Hybrid EPD Disaggregation for Efficient Multimodal Large Language Model Inference \[[Paper](https://openreview.net/forum?id=nNovi8fvGN)]
      * BUAA
      * Propose **TriInfer**, a hybrid encode-prefill-decode disaggregation system for multimodal LLM inference with stage-aware resource management and SLO guarantees.
  * Retrieval-Augmented Generation (RAG)
    * TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval \[[Paper](https://openreview.net/forum?id=YsOyCpMUYD)]
      * UW & CMU & UC Berkeley
      * Propose **TeleRAG**, a lookahead retrieval system that preloads embeddings to reduce RAG latency and improve throughput with minimal GPU memory overhead.
  * Heterogeneous/Edge Inference
    * ExecuTorch - A Unified PyTorch Solution to Run ML Models On-Device \[[Paper](https://openreview.net/forum?id=jmE5nwC9kb)]
      * Meta
      * Introduce **ExecuTorch**, a unified PyTorch-native framework for deploying ML models across heterogeneous edge devices with portable compilation and hardware delegation.
    * Efficient, VRAM-Constrained xLM Inference on Clients \[[Paper](https://openreview.net/forum?id=VKqQYg6JPb)]
      * Intel
      * Enable efficient lossless inference of large language and vision models on VRAM-constrained client systems through adaptive CPU-GPU partitioning.
    * Rethinking DVFS for Mobile LLMs: Unified Energy-Aware Scheduling with CORE \[[Paper](https://openreview.net/forum?id=PSyHQ8kVUT)]
      * CUHK & Purdue & SJTU
      * Propose **CORE**, a unified energy-aware scheduling framework for mobile LLMs that coordinates DVFS decisions across heterogeneous compute units.
    * Locality-Aware Beam Scheduling for Efficient Test-Time Compute with a Consumer-grade GPU \[[Paper](https://openreview.net/forum?id=dTo8jAXm9K)]
      * National Taiwan U
      * Propose locality-aware beam scheduling for efficient test-time compute on consumer GPUs by managing the expanding KV cache during step-wise beam search.
    * SHIP: SRAM-Based Huge Inference Pipelines for Fast LLM Serving \[[Paper](https://openreview.net/forum?id=IZaXDwDtL1)]
      * Groq
      * Present the first large-scale SRAM-based LLM inference deployment (Groq's public cloud) serving hundreds of billions of tokens daily with deterministic latency guarantees.
    * Dataflow Is All You Need \[[Paper](https://openreview.net/forum?id=7wOOhxkuN8)]
      * SambaNova
      * Demonstrate that dataflow architectures achieve superior memory bandwidth utilization for autoregressive LLM decoding compared to GPU-based systems.
  * Profiling
    * ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler \[[Paper](https://openreview.net/forum?id=tYHWS7YPof)]
      * SJTU
      * Develop **ProfInfer**, a non-intrusive eBPF-based profiler providing fine-grained operator-level visibility into LLM inference engines at runtime.
    * Optimizing Deployment Configurations for LLM Inference \[[Paper](https://openreview.net/forum?id=gEbKQeIdxB)]
      * Meta
      * Present a systematic framework for navigating Meta's production LLM deployment configuration space across hardware, parallelism strategies, and runtime choices.
    * DriftBench: Measuring and Predicting Infrastructure Drift in LLM Serving Systems \[[Paper](https://openreview.net/forum?id=Xfzzp6grRP)]
      * Gianluigi Vitale
      * Present **DriftBench**, a measurement and prediction framework for assessing output consistency risks when LLM serving infrastructure changes across hardware, precision, and frameworks.
    * Breaking the Ice: Analyzing Cold Start Latency in vLLM \[[Paper](https://openreview.net/forum?id=eoEobeKTNZ)]
      * VU Amsterdam & TU Darmstadt
      * Present the first detailed performance analysis of cold start latency in vLLM, identifying key bottlenecks in the engine startup process.
    * Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference \[[Paper](https://openreview.net/forum?id=19O6GAS7Su)]
      * Mengtian Yang
      * Introduce **Charon**, a unified modular simulator for accurately predicting LLM training and inference performance across parallelism strategies and hardware configurations.
  * Fault Tolerance
    * GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving \[[Paper](https://openreview.net/forum?id=xKjYiUgeOK)]
      * UCF
      * Present **GhostServe**, a lightweight checkpointing system for fault-tolerant LLM serving that protects long-running agent tasks against hardware and software failures.
    * RaidServe: High-performance Resilient Serving \[[Paper](https://openreview.net/forum?id=5pl9fdbEkq)]
      * Stanford
      * Present **RaidServe**, a fault-tolerant tensor-parallel serving system that sustains high performance under GPU failures through computation and memory rebalancing.
  * Automatic Configuration
    * Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT \[[Paper](https://openreview.net/forum?id=om4H7AI2hc)]
      * Nicholas Santavas
      * Present **OptiKIT**, an automated system for enterprise LLM optimization that enables teams to efficiently deploy models across heterogeneous GPU infrastructure without specialized expertise.
  * Live Inference
    * AIRS: Scaling Live Inference in Resource Constrained Environments \[[Paper](https://openreview.net/forum?id=g1RWik4Gy1)]
      * Microsoft
      * Present **AIRS**, a system for scaling LLM-based live inference for search quality evaluation under resource-constrained environments.
  * Serverless
    * FaaScale: Unlocking Fast LLM Scaling for Serverless Inference \[[Paper](https://openreview.net/forum?id=jgL8LuOVyT)] \[[arXiv](https://arxiv.org/abs/2502.09922)]
      * CUHK-SZ & UVA & HKUST & Alibaba & Nokia Bell Labs
      * Present **FaaScale**, a serverless LLM system enabling fast model scaling through pipelined multicast inference that synergizes data transfer with cross-node pipeline execution.
      * Use **PipeCast** to adaptively multicast model blocks and dynamically form inference pipelines on the fly during model transfer.
  * Multi-model Serving
    * BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization \[[Paper](https://openreview.net/forum?id=ZVQb92umqX)]
      * Cambridge
      * Propose **BOute**, using multi-objective Bayesian optimization for cost-efficient LLM serving across heterogeneous models and GPUs with joint routing and placement.
    * HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving \[[Paper](https://openreview.net/forum?id=CV52m9NJFK)]
      * UT Austin & NVIDIA
      * Propose **HELIOS**, adaptively selecting models and early-exit layers across multiple early-exit LLMs to improve inference throughput beyond single-model serving.
  * Agent Serving
    * FlashAgents: Accelerating Multi-Agent LLM Systems via Streaming Prefill Overlap \[[Paper](https://openreview.net/forum?id=m14PPUfgEc)]
      * ICT, CAS & Microsoft
      * Propose **FlashAgents**, accelerating multi-agent LLM workflows through token-level streaming and prefix-aware prefill overlap between sequential agents.
    * AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents \[[Paper](https://openreview.net/forum?id=UfABxFoSXH)]
      * Stanford
      * Introduce **AgenticCache**, a cache-driven asynchronous planning framework that reuses cached plan transitions to avoid per-step LLM calls for embodied agents.
    * Ontology-Guided Long-Term Agent Memory for Conversational RAG \[[Paper](https://openreview.net/forum?id=wpZHLPz4N0)]
      * Sharon Cao
      * Present an ontology-guided dialogue-aware RAG system that jointly addresses what to store and how to retrieve for long-term multi-session agent memory.
    * Hippocampus: An Efficient and Scalable Memory Module for Agentic AI \[[Paper](https://openreview.net/forum?id=0sUYZh9D4a)]
      * HP Labs
      * Introduce **Hippocampus**, an agentic memory system using compact binary signatures for semantic search and lossless token-ID streams for scalable content storage.
* MoE Training
  * MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs \[[Paper](https://openreview.net/forum?id=L8qKfWWkry)]
    * Meta
    * Propose **MoEBlaze**, breaking the memory wall for MoE training by addressing activation memory overheads from sparse routing buffers and intermediate tensor buffering on GPUs.
  * FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error \[[Paper](https://openreview.net/forum?id=wyH60Su6G7)]
    * Fengjuan Wang
    * Propose **FP8-Flow-MoE**, a casting-free FP8 dataflow for MoE training that eliminates redundant quantize-dequantize conversions while avoiding double quantization error.
  * FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models \[[Paper](https://openreview.net/forum?id=ruOpvLzsGV)]
    * NVIDIA
    * Present **FarSkip-Collective**, modifying MoE architecture with skip connections to enable overlapping computation with blocking all-to-all communication.
* MoE Inference
  * CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving \[[Paper](https://openreview.net/forum?id=zdRvzU9ZCe)]
    * UofT & AWS
    * Propose **CRAFT**, a fine-grained cost-aware expert replication strategy for MoE serving that balances load across devices in large-scale deployments.
  * From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill \[[Paper](https://openreview.net/forum?id=yyDbI3HXco)]
    * SNU
    * Propose layered prefill, redefining stall-free scheduling for MoE serving by splitting along the layer dimension to avoid expert-induced load imbalance.
  * Demystifying the Mixture of Experts Serving Tax \[[Paper](https://openreview.net/forum?id=lELxqcgrsN)]
    * UW
    * Comprehensively study the MoE serving tax, finding MoE models perform 2-3x worse than FLOP-equivalent dense models, and categorize the underlying overhead sources.

### Deep Learning Compilation

* Compiler
  * ApproxMLIR : Accuracy-Aware Compiler for Compound ML System \[[Paper](https://openreview.net/forum?id=nKm25GWbuB)]
    * UIUC
    * Propose **ApproxMLIR**, an accuracy-aware compiler for compound ML systems that enables end-to-end quality-performance tradeoffs across ML and non-ML components.
  * Wave: A Symbolic Python DSL And Compiler for High-Performance Machine Learning \[[Paper](https://openreview.net/forum?id=gcXV1E8HRH)]
    * AMD
    * Introduce **Wave**, a Python-embedded DSL for GPU kernel authoring that automates complex address computations for matrix cores while matching or surpassing state-of-the-art performance.
  * CATWILD: Compiler Autotuning for TPU workloads in the Wild \[[Paper](https://openreview.net/forum?id=hB3nov3gIP)]
    * Google
    * Present **CATWILD**, a compiler autotuning system for diverse production TPU workloads using ML-guided search to optimize configurations at scale.
  * DynaFlow: Transparent and Flexible Intra-Device Parallelism via Programmable Operator Scheduling \[[Paper](https://openreview.net/forum?id=i0yqC9954S)]
    * CMU & UC Berkeley
    * Present **DynaFlow**, enabling transparent intra-device parallelism via programmable operator scheduling without invasive model-specific code changes.
  * Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel \[[Paper](https://openreview.net/forum?id=PJqFhAbUHa)]
    * CMU
    * Present **Event Tensor**, a unified compiler abstraction for dynamic megakernels that handles dynamic shapes and data-dependent computation to eliminate kernel launch overheads.
* Kernel Optimization
  * AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization \[[Paper](https://openreview.net/forum?id=SBS4NJHYjZ)]
    * Stanford & UofT & AWS
    * Present **AccelOpt**, a self-improving LLM agentic system that autonomously optimizes kernels for AI accelerators through iterative generation with optimization memory.
  * Agentic Operator Generation for ML ASICs \[[Paper](https://openreview.net/forum?id=O3Bx0nNGnW)]
    * Meta
    * Present **TritorX**, an agentic AI system that generates correct Triton kernels at scale for Meta's MTIA accelerator using LLMs with compilation-based validation.
  * Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems \[[Paper](https://openreview.net/forum?id=MJxhiX3sSd)]
    * LBNL
    * Study LLM-based multi-agent systems for GPU kernel optimization, analyzing dynamics and effectiveness compared to existing compilers and manual development.
* Attention
  * FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling \[[Paper](https://openreview.net/forum?id=mN5RtvuYl3)]
    * Together AI & NVIDIA
    * Propose **FlashAttention-4**, co-designing algorithm and kernel pipelining for Blackwell GPUs where tensor core throughput growth outpaces memory bandwidth.
  * Flashlight: PyTorch Compiler Extensions to Accelerate Attention Variants \[[Paper](https://openreview.net/forum?id=lboOMA8XWr)]
    * UW & UT Austin
    * Present **Flashlight**, PyTorch compiler extensions that automatically generate efficient implementations for diverse attention variants through composable primitives.
  * IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference \[[Paper](https://openreview.net/forum?id=CPCRITwAaP)]
    * SUSTech
    * Propose **IntAttention**, a fully integer attention pipeline eliminating the costly dequantize-softmax-requantize detour for end-to-end integer dataflow on edge hardware.

### Diffusion / Generative Models

* StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation \[[Paper](https://openreview.net/forum?id=p9WALNBvc6)]
  * MIT & UC Berkeley & Stanford
  * Present **StreamDiffusionV2**, a streaming system for real-time dynamic and interactive video generation with temporal consistency built on video diffusion advances.
* CDLM: Consistency Diffusion Language Models for Faster Sampling \[[Paper](https://openreview.net/forum?id=eB8yjR6alL)]
  * UC Berkeley & Together AI
  * Introduce **CDLM**, applying consistency modeling to diffusion language models for faster sampling with fewer refinement steps and KV caching support.
* db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism \[[Paper](https://openreview.net/forum?id=XgKteNxNe0)]
  * THU
  * Propose **db-SP**, a dual-balanced sequence parallelism scheme for accelerating sparse attention in DiT models with improved workload balance across head and block dimensions.

### GPU Systems

* GPU Programming
  * HipKittens: Fast and Furious AMD Kernels \[[Paper](https://openreview.net/forum?id=xxSSrndQrI)]
    * Stanford & AMD
    * Port ThunderKittens to AMD GPUs via HIP, enabling high-performance tile-based AI kernel development on AMD hardware with hardware-aware primitives.
  * ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels \[[Paper](https://openreview.net/forum?id=Cv5e5uRXFb)]
    * Stanford
    * Propose **ParallelKittens**, a set of simple reusable principles for systematically designing multi-GPU kernels with optimal compute-communication overlap.
  * Spira: Exploiting Voxel Data Structural Properties for Efficient Sparse Convolution in Point Cloud Networks \[[Paper](https://openreview.net/forum?id=YQMilw805Q)]
    * NTUA & UofT
    * Propose **Spira**, exploiting voxel data structural properties (integer-valued, bounded, geometrically continuous) for efficient sparse convolution in point cloud networks.
* Profiling
  * XProf: An Open, Scalable, and Extensible Profiling System for the Modern ML Stack \[[Paper](https://openreview.net/forum?id=KqRLAdGK6C)]
    * Google
    * Present **XProf**, an open ML profiler for the OpenXLA ecosystem providing actionable optimization suggestions and full-stack performance analysis across TPUs and GPUs.
  * Hawkeye: Reproducing GPU-Level Non-Determinism \[[Paper](https://openreview.net/forum?id=JnmgsTFQQv)]
    * Stanford & Hebrew U
    * Present **Hawkeye**, a system for reproducing GPU-level arithmetic operations on CPU to analyze and verify non-determinism in ML training and inference workflows.
* Confidential Computing
  * Blueprint, Bootstrap, and Bridge: A Security Look at NVIDIA GPU Confidential Computing \[[Paper](https://openreview.net/forum?id=t9RDCO1aL7)]
    * IBM Research & Ohio State
    * Provide the first comprehensive security analysis of NVIDIA GPU Confidential Computing by reconstructing the architecture from firmware and driver analysis.

### Communication

* A Lightweight High-Throughput Collective-Capable NoC for Large-Scale ML Accelerators \[[Paper](https://openreview.net/forum?id=VDuS8N9RCx)]
  * ETH Zurich
  * Present a lightweight collective-capable Network-on-Chip supporting efficient barrier, broadcast, reduce, and scatter operations for large-scale on-chip ML accelerator arrays.
* fabric-lib: RDMA Point-to-Point Communication for LLM Systems \[[Paper](https://openreview.net/forum?id=SjVa05wEiY)]
  * Nandor Licker
  * Present **fabric-lib**, a uniform RDMA-based messaging layer bridging diverse NICs to provide portable point-to-point communication for disaggregated LLM inference and MoE routing.

### Resource Management

* Machine Learning Fleet Efficiency: Improving TPU Systems at Scale with ML Productivity Goodput \[[Paper](https://openreview.net/forum?id=y31QSL9yMG)]
  * Google & Harvard
  * Introduce a systematic framework for analyzing ML fleet efficiency on Google's production TPU infrastructure, revealing performance dependencies across the full system stack.
* SAKURAONE: An Open Ethernet–Based AI HPC System and Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment \[[Paper](https://openreview.net/forum?id=n7o6C3p3wk)]
  * SAKURA Internet
  * Present **SAKURAONE**, an open Ethernet-based AI HPC system (TOP500 #49) using vendor-neutral 800 GbE with SONiC networking, with workload analysis from single-tenant LLM training.

### Recommendation Systems

* FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost \[[Paper](https://openreview.net/forum?id=MY0BIdK4hn)]
  * Meta
  * Introduce **FreeScale**, a distributed training system for sequence recommendation models that minimizes scaling cost by addressing computational bubbles from data heterogeneity and straggler imbalance.

## Acronyms

* BFP: Block Floating Point
* CP: Context Parallelism
* DiT: Diffusion Transformer
* FSDP: Fully Sharded Data Parallel
* KV: Key-Value
* LLM: Large Language Model
* LoRA: Low-Rank Adaptation
* MoE: Mixture-of-Experts
* NoC: Network-on-Chip
* PD: Prefill-Decode
* QAT: Quantization-Aware Training
* RAG: Retrieval-Augmented Generation
* RL: Reinforcement Learning
* SLO: Service Level Objective
* TPU: Tensor Processing Unit


# NSDI 2026

## Meta Info

Homepage: <https://www.usenix.org/conference/nsdi26>

Paper list: <https://www.usenix.org/conference/nsdi26/technical-sessions>

### Acceptance Rate

* Spring: 24.2% (= 50 / 207)
* Fall: 22.1% (= 100 / 452)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * MoE Training
    * SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/skiadopoulos)] \[[arXiv](https://arxiv.org/abs/2504.19925)]
      * Stanford & NVIDIA & OpenAI
      * Propose **SYMI**, an adaptive MoE training system that decouples the placement of expert parameters from their large optimizer states.
      * Statically partition optimizer states across training nodes while dynamically adjusting expert parameter placement using existing weight updates, avoiding frequent state migration overheads.
      * Improve time-to-convergence by 30.5% over DeepSpeed and 25.9% over FlexMoE.
    * Checkpoint Lite, Recover Right: Efficient Fault Tolerant Training of Mixture-of-Experts Models Using Sparse Checkpoints \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/gandhi)]
      * Stanford & NVIDIA
  * Cross-Cluster Training
    * Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-Cluster LLM Training at Scale \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/li-shengwei)]
      * NUDT & Shanghai AI Lab & NTU
  * Fault Tolerance
    * Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/wu-tianyuan)] \[[arXiv](https://arxiv.org/abs/2504.19232)]
      * HKUST & Alibaba
      * Present a straggler-resilient hybrid-parallel training system for pipeline-parallel large-model training under communication stragglers.
      * Adapt the pipeline schedule with an analytical model to absorb slow communication without cascading bubbles, and offload delayed communication to host memory with CPU-side RDMA to avoid GPU head-of-line blocking.
      * Reduce training iteration time by 1.2x to 3.5x under various straggler settings.
    * Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/cui)] \[[arXiv](https://arxiv.org/abs/2502.05413)]
      * SJTU & Ant Group & NUS
      * Introduce **Flare**, a diagnostic framework for distributed LLM training at scale.
      * Combine a lightweight tracing daemon for full-stack, backend-extensible tracing with a diagnostic engine that automatically diagnoses anomalies, especially performance regressions.
      * Demonstrate deployment across 6,000 GPUs with continuous operation for over eight months in production scenarios.
    * EROICA: Online Performance Troubleshooting for Large-scale Model Training in Production \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/guan-yu)] \[[arXiv](https://arxiv.org/abs/2506.08528)]
      * Alibaba
      * Present **EROICA**, an online troubleshooting system for large-scale model training that combines fine-grained profiling with full-cluster coverage.
      * Summarize runtime execution patterns through online profiling and use differential observability to localize hardware, software, and mixed root causes with minimal production impact.
      * Report deployment on production GPU clusters of about 100,000 GPUs for 1.5 years, diagnosing difficult performance issues with 97.5% success.
  * RL Post-Training
    * RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/gao-wei)] \[[arXiv](https://arxiv.org/abs/2509.21009)]
      * HKUST & Alibaba
      * Introduce tail batching, a rollout scheduling strategy for synchronous RL post-training that packs prompts with long-tail responses into a small number of long rounds while keeping most rounds balanced and short.
      * Combine tail batching with elastic rollout parallelism, dynamic reward-stage resource scheduling, and stream-based training to reduce rollout bubbles without relaxing synchronization.
      * Cut end-to-end training time by 2.03x to 2.56x over veRL and by up to 2.24x over RLHFuse on Qwen2.5 models.
  * Performance Modeling and Simulation
    * Supercharging Packet-Level Network Simulation of Large Model Training via Memoization and Fast-Forwarding \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/long)] \[[arXiv](https://arxiv.org/abs/2602.10615)]
      * THU & Zhongguancun Laboratory & BUPT & Huawei
      * Propose **Wormhole**, a user-transparent packet-level discrete-event simulation kernel for large-model training that reduces redundant simulation work without simplifying the network model.
      * Memoize unsteady states and fast-forward steady states through network partitioning, state reuse, and rate-based steady-state identification while preserving simulation consistency.
      * Achieve 744x speedup over ns-3 with less than 1% error; combined with multithreading, the speedup reaches 1012x.
    * GPUSynth: Maximizing Code Reuse in Simulation-Based Machine Learning System Performance Estimation \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/qin)] \[[arXiv](https://arxiv.org/abs/2505.01616)]
      * Duke & UC Berkeley & Meta
      * Introduce **Phantora**, a hybrid GPU-cluster simulator for ML training performance estimation that runs unmodified ML frameworks in a distributed containerized environment.
      * Intercept GPU and communication operations during execution, enabling direct reuse of ML framework source code instead of reimplementing frameworks inside a simulator.
      * Match the accuracy of static workload simulation while supporting three state-of-the-art LLM training frameworks out of the box on a single GPU.
* LLM Inference
  * Request Scheduling
    * FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/wu-bingyang)] \[[arXiv](https://arxiv.org/abs/2305.05920)] \[[Code](https://github.com/LLMServe/FastServe)]
      * PKU
      * Propose **FastServe**, an LLM serving system that enables iteration-level preemptive scheduling for autoregressive decoding instead of request-level FIFO execution.
      * Combine a skip-join multi-level feedback queue scheduler with a parallelism-aware suspension strategy that downscales preempted models into CPU/NVMe storage.
      * Improve multi-model serving throughput and latency; the abstract reports up to 5.1x higher throughput and 13.9x to 111.8x lower latency than prior systems.
    * JITServe: SLO-aware LLM Serving with Imprecise Request Information \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/zhang-wei)] \[[arXiv](https://arxiv.org/abs/2504.20068)]
      * UIUC & NJU & Google Labs & Cisco Research
      * Propose **JITServe**, an LLM serving system for workloads where request information is imprecise at arrival time.
      * Forecast future decoding lengths from near-future token generation and construct just-in-time execution schedules that balance prediction accuracy against rapidly changing system dynamics.
      * Increase throughput while improving latency SLO satisfaction; the abstract reports 1.8x to 7.5x higher throughput and 2.2x to 8.7x better SLO attainment than prior systems.
    * Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/ruan-libra)]
      * NUS & USTC & UC Berkeley
  * KV Cache Management
    * DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/liu-yuhan)] \[[arXiv](https://arxiv.org/abs/2411.02820)]
      * UChicago & Microsoft
      * Introduce **DroidSpeak**, the first distributed LLM inference system that reuses prefix KV caches across different LLMs with the same architecture, including across distributed nodes.
      * Selectively recompute a small subset of layers from another model's KV cache and reuse the remaining layers, then pipeline recomputation with reused-cache loading to improve performance while preserving quality.
      * Improve throughput by up to 4x and prefill latency by about 3.1x with negligible quality loss.
    * SYMPHONY: Improving Memory Management for LLM Inference Workloads \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/agarwal)] \[[arXiv](https://arxiv.org/abs/2412.16434)]
      * UT Austin & UW-Madison
      * Exploit hints from multi-turn workloads to migrate KV caches off the critical serving path instead of recomputing them or pinning serving to specific machines through host-memory offload.
      * Dynamically migrate KV caches to enable fine-grained scheduling of inference requests across the cluster.
      * Handle more than 8x as many requests as state-of-the-art baselines while preserving a similar latency profile.
  * Workload Characterization
    * Seshat: Workload Characterization and Generation of Large Language Model Serving in Production \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/xiang-servegen)] \[[arXiv](https://arxiv.org/abs/2505.09999)]
      * PKU & Alibaba
      * Characterize production LLM serving workloads from a worldwide cloud inference service, covering language, multimodal, and reasoning models.
      * Build a per-client workload generation framework that composes realistic serving workloads from the observed production characteristics.
      * Avoid 50% under-provisioning compared with naive workload generation in a production validation case.
  * Serverless Computing
    * HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/lou)] \[[arXiv](https://arxiv.org/abs/2502.15524)] \[[Code](https://github.com/LLMServe/hydraserve)]
      * PKU & Alibaba Cloud
      * Present **HydraServe**, a serverless LLM serving system for public clouds that minimizes cold-start latency.
      * Proactively distribute models across servers, overlap cold-start stages within workers, place workers to avoid GPU-network contention, and consolidate pipelines to reduce cold-start resource usage.
      * Reduce cold-start latency by 1.7x to 4.7x and improve SLO attainment by 1.43x to 1.74x over baselines.
  * Multiplexing
    * FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/oliaro)] \[[arXiv](https://arxiv.org/abs/2402.18789)]
      * CMU & Purdue & Anthropic & Mistral AI & Stanford
      * Introduce **FlexLLM**, the first system to co-serve LLM inference and parameter-efficient fine-tuning on shared GPUs by fusing computation at the token level.
      * Use dependent parallelization and graph pruning to shrink activation memory, then interleave inference and training tokens with token-level finetuning and a hybrid token scheduler to meet latency SLOs.
      * Save up to 80% of GPU memory and improve finetuning throughput by 1.9x to 4.8x under heavy inference load and 2.5x to 6.8x under light load.
  * LLM Agents
    * Agentix: An Efficient Serving Engine for LLM Agents as General Programs \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/luo)] \[[arXiv](https://arxiv.org/abs/2502.13965)]
      * UC Berkeley & Google DeepMind & SJTU
      * Treat agent programs as first-class scheduling entities in LLM serving to reduce end-to-end latency for agentic workloads.
      * Intercept program-issued LLM calls to expose program-level context and preemptively prioritize calls based on previously completed work for both single-threaded and distributed programs.
      * Improve program throughput by 4x to 15x at the same latency compared with systems such as vLLM.
    * Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge Caching \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/ruan-cortex)] \[[arXiv](https://arxiv.org/abs/2509.17360)]
      * NUS & USTC & UofT & Sea AI Lab
      * Introduce **Cortex**, a cross-region knowledge caching architecture for LLM agents that targets semantic reuse rather than exact-match query reuse.
      * Build semantic-aware retrieval on Semantic Element and Semantic Retrieval Index abstractions, then add semantic-aware cache hits, eviction, prefetching, and a colocated lightweight LLM judger.
      * Increase throughput by up to 3.6x on search workloads while preserving accuracy close to uncached baselines, and improve coding-task throughput by 20%.
  * MoE Inference
    * SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/li-xingyi)]
      * Tencent & Nanjing University & Zhongguancun Laboratory
* LLM Fine-Tuning
  * MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/xue-chunyu)] \[[arXiv](https://arxiv.org/abs/2603.02885)]
    * SJTU & NUS
    * Present **MuxTune**, a multi-tenant fine-tuning system that spatially and temporally multiplexes the shared backbone across concurrent parameter-efficient fine-tuning tasks.
    * Build on unified fine-tuning representations with hierarchical co-scheduling across tasks, operators, and data, including hybrid spatial-temporal multiplexing, two-tier hybrid parallelism, and chunk-based data alignment.
    * Achieve up to 2.33x higher throughput and 5.29x lower memory usage than prior baselines.
* LLM Storage
  * ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/wang-zirui)] \[[arXiv](https://arxiv.org/abs/2505.06252)] \[[Homepage](https://storageai.github.io/ZLLM/)]
    * UVA & Harvard
    * Characterize all publicly available Hugging Face LLM repositories and identify structured sparse deltas within model families, bitwise-similarity-based family clustering, and tensor-level deduplication as the right storage granularity.
    * Design **BitX**, a lossless delta compression algorithm for XORed differences between fine-tuned and base models, and build **ZipLLM** to unify tensor-level deduplication with BitX compression.
    * Reduce model storage consumption by 54%, over 20% better than prior deduplication and compression approaches.

### Distributed Training

* Checkpointing
  * Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/bhardwaj)] \[[arXiv](https://arxiv.org/abs/2507.13522)]
    * Tufts & MIT
    * Introduce **Checkmate**, a model checkpointing system that reuses network-replicated gradients to avoid additional network transfer and disk I/O overhead on the normal checkpointing path.
    * Use dynamically reconfigurable in-network checkpoint replica placement and a failure-resilient accelerator pipeline to preserve resilience under failures while overlapping checkpoint creation with gradient computation and propagation.
    * Report nearly 100% throughput improvement over GPU-optimized checkpointing and up to 13.7% over asynchronous checkpointing on a 32-node GPU cluster.
* Collective Communication
  * FAST: An Efficient Scheduler for All-to-All GPU Communication \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/lei-yiran)] \[[arXiv](https://arxiv.org/abs/2505.09764)]
    * CMU & MangoBoost & UW & UPenn
    * Present **FAST**, an efficient All-to-All(v) scheduler for modern ML workloads, especially MoE models on heterogeneous two-tier GPU fabrics.
    * Address workload skew through intra-server rebalancing and enforce balanced one-to-one scale-out transfers to avoid incast congestion.
    * Outperform prior schedulers on skewed workloads while reducing schedule synthesis time by orders of magnitude.
  * HeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous GPU Clusters \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/hei)]
    * Northeastern University & Alibaba Cloud & SIAT, CAS
  * ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics \[[Paper](https://www.usenix.org/conference/nsdi26/presentation/zhao-liangyu)] \[[arXiv](https://arxiv.org/abs/2402.06787)]
    * UW & THU & Microsoft
    * Present **ForestColl**, a schedule generation tool that constructs broadcast and aggregation spanning trees to produce throughput-optimal collective communication schedules for arbitrary network topologies.
    * Achieve theoretical optimality with polynomial-time schedule generation while supporting both switching fabrics and direct accelerator interconnects.
    * Outperform vendor communication libraries and prior schedule generation methods on AMD MI250 and NVIDIA DGX A100 and H100 clusters, including LLM training workloads.

## Acronyms

* KV: Key-Value
* LLM: Large Language Model
* MoE: Mixture-of-Experts
* RL: Reinforcement Learning
* SLO: Service Level Objective


# EuroSys 2026

## Meta Info

Homepage: <https://2026.eurosys.org>

Paper list: <https://2026.eurosys.org/papers.html>

### Acceptance Rate

* Spring: 19.6% (= 79 / 404)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * MoE Training
    * MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production \[[Paper](https://doi.org/10.1145/3767295.3769325)] \[[arXiv](https://arxiv.org/abs/2505.11432)]
      * PKU & ByteDance
      * Present **MegaScale-MoE**, a production system for efficient large-scale MoE training.
      * Co-design communication-efficient parallelism, inter- and intra-operator communication-computation overlap, and lower-precision communication compression for MoE layers.
  * LoRA Fine-Tuning
    * LoRAFusion: Efficient LoRA Fine-Tuning for LLMs \[[Paper](https://doi.org/10.1145/3767295.3769331)] \[[arXiv](https://arxiv.org/abs/2510.00206)]
      * UofT & Vector Institute & NVIDIA
      * Present **LoRAFusion**, a system that improves LoRA fine-tuning by optimizing both fused kernels and multi-job training schedules.
      * Combine graph-splitting-based kernel fusion with multi-job adaptive batching to reduce memory traffic, improve communication overlap, and mitigate pipeline bubbles.
    * Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices \[[Paper](https://doi.org/10.1145/3767295.3769329)]
      * Shandong University & XJTU
      * Study federated fine-tuning for sparsely activated LLMs under resource-constrained client devices.
  * Data Pipeline
    * MegaScale-Data: Scaling DataLoader for Multisource Large Foundation Model Training \[[Paper](https://doi.org/10.1145/3767295.3803568)] \[[arXiv](https://arxiv.org/abs/2504.09844)]
      * HKU & ByteDance
      * Present **MegaScale-Data**, an industrial-grade distributed data loading architecture for multi-source large foundation model training.
      * Disaggregate preprocessing with role-specific actors and use a centralized declarative data plane to support scalable multi-source orchestration under heterogeneous preprocessing costs.
  * Scheduling and Parallelism
    * STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planning \[[Paper](https://doi.org/10.1145/3767295.3769335)]
      * THU & Infinigence-AI & SJTU
      * Plans memory usage across spatial and temporal dimensions to improve large-scale model training efficiency.
    * Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training \[[Paper](https://doi.org/10.1145/3767295.3769369)]
      * PKU & ETH & CUHK & Shanghai AI Lab & MIT
      * Balances variable-length data-parallel training workloads to reduce stragglers and improve utilization.
    * Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design \[[Paper](https://doi.org/10.1145/3767295.3803571)]
      * SJTU & Lenovo Research & Microsoft & Guizhou University & NUS
      * Co-designs dynamic scheduling and adaptive parallelism for efficient large-model training.
    * HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters \[[Paper](https://doi.org/10.1145/3767295.3803603)]
      * Fudan & Shandong Computer Science Center
      * Automates parallel training orchestration across heterogeneous GPU clusters.
    * HetAuto: Cross-Cluster Auto-Parallelism for Heterogeneous Distributed Training \[[Paper](https://doi.org/10.1145/3767295.3803590)]
      * HKU & Meituan
      * Searches cross-cluster auto-parallel strategies for heterogeneous distributed training environments.
    * Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering \[[Paper](https://doi.org/10.1145/3767295.3769370)]
      * THU & PKU & Infinigence-AI & SJTU
      * Uses signaling and reordering to adaptively overlap computation and communication.
    * Crimson: Collaborative Parameter Updates for Efficient Pipeline Training of Large Language Models \[[Paper](https://doi.org/10.1145/3767295.3803606)]
      * SYSU & HKUST & Pengcheng Laboratory
      * Coordinates parameter updates to improve pipeline training efficiency for large language models.
    * Suika: Efficient and High-quality Re-scheduling of 3D-parallelized LLM Training Jobs in Shared Clusters \[[Paper](https://doi.org/10.1145/3767295.3803623)]
      * SJTU & TeleAI & Huawei
      * Re-schedules 3D-parallel LLM training jobs in shared clusters while preserving training quality.
  * Runtime Modeling
    * Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation \[[Paper](https://doi.org/10.1145/3767295.3769366)] \[[arXiv](https://arxiv.org/abs/2503.20191)]
      * Georgia Tech & NVIDIA
      * Present **Maya**, a performance modeling system for deep learning training based on transparent GPU device emulation.
      * Intercept device API calls from unmodified training code to observe low-level operations without workload translation or code modification.
  * Multimodal Training
    * MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production \[[Paper](https://doi.org/10.1145/3767295.3803587)] \[[arXiv](https://arxiv.org/abs/2605.08962)]
      * SJTU & ByteDance
      * Present **MegaScale-Omni**, a production multimodal LLM training system for dynamic modality mixtures and sample-length distributions.
      * Combine decoupled encoder/backbone parallelism, encoder-LLM joint pipelines, and workload balancing for large-scale MLLM training.
  * Fault Tolerance
    * Handling Network Faults in Distributed AI Training: Failover is Now an Option \[[Paper](https://doi.org/10.1145/3767295.3769322)]
      * NUS & ByteDance
      * Revisit network-fault handling in distributed AI training and make failover practical for large jobs.
* LLM Inference
  * Speculative Decoding
    * AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding \[[Paper](https://doi.org/10.1145/3767295.3769315)]
      * CMU & Princeton & EPFL & AWS & Purdue
      * Customize speculative decoding policies for serving workloads with multiple latency SLOs.
  * Request Scheduling
    * FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters \[[Paper](https://doi.org/10.1145/3767295.3769316)] \[[arXiv](https://arxiv.org/abs/2510.11938)]
      * SIAT, CAS & UCAS & UCSD & University of Macau
      * Refactors in-flight pipelines to adapt LLM serving to fragmented serverless GPU clusters.
    * TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling \[[Paper](https://doi.org/10.1145/3767295.3769328)]
      * SJTU & GMU & China Telecom Shanghai
      * Applies preemptive scheduling to keep text-streaming LLM serving responsive under request bursts.
    * AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving \[[Paper](https://doi.org/10.1145/3767295.3769345)]
      * UVA & HPE Labs & UC Riverside
      * Adapts cluster scheduling decisions to workload shifts for latency-optimized LLM inference.
    * SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference \[[Paper](https://doi.org/10.1145/3767295.3769353)]
      * UC Berkeley & RUC & Rice
      * Uses locality-aware cross-region load balancing for distributed LLM inference services.
    * PiLLM: Resource-Efficient LLM Inference Using Workload Prediction \[[Paper](https://doi.org/10.1145/3767295.3769393)]
      * ShanghaiTech & SenseTime & BUAA
      * Predicts workload behavior to improve resource efficiency in LLM inference.
  * KV Cache and Memory Management
    * Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading \[[Paper](https://doi.org/10.1145/3767295.3769319)] \[[arXiv](https://arxiv.org/abs/2502.05370)]
      * Stevens Institute of Technology & Waterloo & Rutgers
      * Uses fine-grained expert offloading to balance latency and memory pressure in MoE-based LLM serving.
    * KUNSERVE: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving \[[Paper](https://doi.org/10.1145/3767295.3769348)]
      * SJTU
      * Manages model parameters as the central memory object to handle memory overloads during LLM serving.
    * High Throughput and Low Latency LLM Serving via Adaptive KV Caching \[[Paper](https://doi.org/10.1145/3767295.3803570)]
      * University of Macau & SIAT, CAS & NTU
      * Adapts KV caching decisions to jointly improve throughput and latency.
  * Multiplexing
    * MFS: An Efficient Model Family Serving System for LLMs \[[Paper](https://doi.org/10.1145/3767295.3769355)]
      * HKUST & USTC & Inspur
      * Serves related LLM model families by exploiting shared model structure and deployment opportunities.
    * Efficient Multimodal Serving via Module Multiplexing \[[Paper](https://doi.org/10.1145/3767295.3769389)]
      * HKUST & SYSU & XJTU & MetaX
      * Multiplexes reusable multimodal model modules to improve serving efficiency.
  * Sparsity
    * SAS: Sparse Attention Synthesizer for Efficient Language Model Inference \[[Paper](https://doi.org/10.1145/3767295.3769364)]
      * Amazon
      * Synthesizes sparse attention patterns to accelerate language-model inference.
  * Heterogeneous Deployment
    * Scaling LLM Test-Time Compute with Mobile NPU on Smartphones \[[Paper](https://doi.org/10.1145/3767295.3769382)]
      * THU & USTC & MSR & AIR, THU
      * Explores using mobile NPUs to scale LLM test-time compute on smartphones.
    * TailorLLM: Collaborative End-Cloud Inference of Large and Small Language Models Based on Low-Rank Adaptation \[[Paper](https://doi.org/10.1145/3767295.3769346)]
      * BUPT
      * Coordinates cloud and edge execution between large and small language models using low-rank adaptation.
  * Trusted Execution
    * TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone \[[Paper](https://doi.org/10.1145/3767295.3769334)]
      * SJTU
      * Protects on-device LLM execution with Arm TrustZone-based isolation.
  * LLM-based Applications
    * AIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments \[[Paper](https://doi.org/10.1145/3767295.3803622)]
      * UVA & Microsoft
      * Places and executes LLM-based agents across hybrid cloud-edge environments with cost efficiency.
    * From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents \[[Paper](https://doi.org/10.1145/3767295.3803576)]
      * IS, CAS & UCAS & SJTU
      * Recasts OS interfaces from imperative actions toward declarative forms that are easier for computer-use agents to operate.

### Diffusion Models

* Image Editing
  * FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling \[[Paper](https://doi.org/10.1145/3767295.3769379)] \[[arXiv](https://arxiv.org/abs/2505.20600)] \[[Code](https://github.com/Sylvia-16/FlashPS)]
    * HKUST & Alibaba
    * **Our work!**
    * Uses mask-aware caching and scheduling to accelerate generative image editing workloads.

### Model Serving

* Automated End-to-End Model Serving with Cooperative Compilation and Scheduling \[[Paper](https://doi.org/10.1145/3767295.3769392)]
  * NJU & Hunan University
  * Jointly optimizes model-serving compilation and scheduling decisions across the end-to-end serving stack.

### Resource Management

* Serverless Computing
  * Efficient Data Passing for Serverless Inference Workflows: A GPU-Centric Approach \[[Paper](https://doi.org/10.1145/3767295.3769336)]
    * HUST & CUHK-Shenzhen & TeleAI & HKUST
    * Designs GPU-centric data passing for serverless inference workflows to reduce transfer overhead.
  * iRoute: Local Routing Table-based Workflow Management in Serverless Computing \[[Paper](https://doi.org/10.1145/3767295.3769318)]
    * TJU & THU & IEIT Systems & Inspur
    * Uses local routing-table mechanisms to manage serverless workflows.
  * DROPS: Managing Serverless Resource Pools in Microsoft Azure Functions \[[Paper](https://doi.org/10.1145/3767295.3769350)]
    * Waterloo & MSR & Microsoft
    * Manages serverless resource pools in Azure Functions to improve allocation efficiency.
  * Squeezy: Rapid VM Memory Reclamation for Serverless Functions \[[Paper](https://doi.org/10.1145/3767295.3769357)]
    * NTUA & UIUC
    * Reclaims VM memory quickly for serverless functions to improve density and responsiveness.
  * Demystifying Serverless Costs on Public Platforms: Bridging Billing, Architecture, and OS Scheduling \[[Paper](https://doi.org/10.1145/3767295.3769374)]
    * UBC & Johns Hopkins
    * Connects public-platform billing behavior with architecture and OS scheduling effects in serverless systems.
  * Fix: externalizing network I/O in serverless computing \[[Paper](https://doi.org/10.1145/3767295.3769387)]
    * Stanford
    * Externalizes network I/O from serverless functions to reduce runtime overhead and simplify function execution.
* GPU Cluster Management
  * Bridging the GPU Utilization Gap: Predictive Multi-Dimensional Resource Scheduling for AI Workloads \[[Paper](https://doi.org/10.1145/3767295.3803579)]
    * THU & Alibaba & SJTU
    * Uses predictive multidimensional scheduling to improve GPU utilization for AI workloads.
  * Untangling GPU Power Consumption: Job-Level Inference in Cloud Shared Settings \[[Paper](https://doi.org/10.1145/3767295.3769333)] \[[PDF](https://hal.science/hal-05291033v1/file/GPU_power_Eurosys.pdf)]
    * ETS & Inria & OVHcloud & CNRS
    * Present practical job-level power estimation methods for GPUs under temporal sharing, spatial sharing, and passthrough deployment modes in cloud environments.
    * Show that GPU sharing can improve energy efficiency for small AI workloads, and identify substantial GPU underutilization in an IaaS GPU cluster.

## Acronyms

* LLM: Large Language Model
* LoRA: Low-Rank Adaptation
* MoE: Mixture-of-Experts


# ASPLOS 2026

## Meta Info

Homepage: <https://www.asplos-conference.org/asplos2026/>

### Paper List

* Program: <https://www.asplos-conference.org/asplos2026/program/>
* Proceedings Volume 1: <https://dl.acm.org/doi/proceedings/10.1145/3760250>
* Proceedings Volume 2: <https://dl.acm.org/doi/proceedings/10.1145/3779212>

### Acceptance Rate

* Review model: ASPLOS 2026 used two submission cycles (`Spring` and `Summer`) and retained a `Major Revision` path for selected papers.
* Spring Cycle: 9.6% (= 20 / 208)
  * Major Revision: 9.1% (= 19 / 208)
* Summer Cycle: 15.7% (= 132 / 840)
* Total: 14.5% (= 152 / 1048)

## Papers

### Large Language Models (LLMs)

* LLM Inference
  * Prefill-Decode Multiplexing
    * Towards High-Goodput LLM Serving with Prefill-decode Multiplexing \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790236)] \[[arXiv](https://arxiv.org/abs/2504.14489)]
      * SJTU & HKU & NUS
      * Propose **MuxWise**, an LLM serving framework built on intra-GPU prefill-decode multiplexing.
      * Integrate a bubble-less multiplex engine, a contention-tolerant estimator, and an SLO-aware dispatcher.
    * Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790135)] \[[arXiv](https://arxiv.org/abs/2504.19516)] \[[Code](https://github.com/zejia-lin/Bullet)]
      * SYSU
      * Enable concurrent execution of prefill and decode requests.
      * Dynamically provision GPU resources based on real-time performance modeling.
    * TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790237)]
      * PKU & Tencent YouTu Lab
      * Introduce tensor-parallel latent attention for disaggregated prefill/decode inference.
      * Combine latent attention with tensor parallelism to improve PD-disaggregated long-context serving.
  * Scheduling
    * QoServe: Breaking the Silos of LLM Inference Serving \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790206)] \[[arXiv](https://arxiv.org/abs/2503.22562)]
      * MSR India
      * Introduce fine-grained QoS classification so applications can specify precise latency requirements, and adapt scheduling decisions to real-time system state.
      * Leverage the predictable execution characteristics of LLM inference to implement dynamic chunking for higher throughput under strict QoS guarantees.
      * Combine hybrid prioritization with selective request relegation to balance fairness, efficiency, and graceful degradation under overload.
    * Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790219)] \[[arXiv](https://arxiv.org/abs/2509.16495)]
      * Snowflake
      * Introduce **Shift Parallelism**, a runtime that switches across inference parallelism strategies for dynamic workloads.
      * Turn parallelism selection into a runtime control decision to jointly improve latency and throughput.
    * XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762228)]
      * Huawei & THU & Shanghai AI Lab
      * Present **XY-Serve**, an end-to-end serving system for dynamic production LLM workloads.
      * Coordinate scheduling, batching, and runtime resource management to sustain serving efficiency under workload variation.
    * BlendServe: Optimizing Offline Inference with Resource-Aware Batching \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790133)]
      * UC Berkeley & UW & UC Davis & Rice
      * Present a resource-aware batching framework for offline inference.
      * Form batches against actual compute and memory bottlenecks to improve throughput.
  * MoE Inference
    * MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790187)]
      * SJTU & CUHK
      * Introduce an MoE inference system with adaptive-precision expert offloading.
      * Jointly tune expert offloading and precision to reduce memory pressure during serving.
  * Compression
    * ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790250)] \[[arXiv](https://arxiv.org/abs/2603.17435)] \[[Code](https://github.com/xxyux/ZipServ)]
      * HKUST-GZ & HIT-SZ & HKUST
      * Introduce hardware-aware lossless compression for LLM inference.
      * Reduce memory footprint while preserving exact model behavior and improving serving efficiency.
  * Speculative Decoding
    * DFVG: A Heterogeneous Architecture for Speculative Decoding with Draft-on-FPGA and Verify-on-GPU \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790153)]
      * SJTU & Eastern Institute of Technology, Ningbo & Southeast University & Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
      * Propose a heterogeneous speculative decoding architecture with FPGA draft generation and GPU verification.
      * Pipeline draft and verify across devices to reduce end-to-end decoding latency.
    * SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790246)]
      * ByteDance Seed & UChicago
      * Introduce disaggregated speculative decoding together with fused kernels for low-latency LLM inference.
      * Combine system-level disaggregation and kernel-level optimization to make speculative decoding practical in deployment.
  * Sparsity
    * SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790224)]
      * SJTU & Infinigence-AI & SII & THU
      * Introduce speculative context sparsity for long-context reasoning in LLMs.
      * Avoid uniform full-context processing by speculating over sparse context usage during long-input inference.
  * Attention Mechanisms
    * I/O Analysis is All You Need: An I/O Analysis for Long-Sequence Attention \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790174)]
      * IIT & ICT, CAS & UCAS
      * Present an I/O-centric analysis framework for long-sequence attention.
      * Show that data movement, rather than FLOPs alone, dominates long-context attention cost.
    * PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790200)]
      * TJU & Stevens Institute of Technology
      * Introduce prefix-aware attention together with a multi-tile kernel for LLM decoding.
      * Reduce decode latency by exploiting shared prefixes while keeping GPU resource usage under control.
  * Value Level Parallelism (VLP)
    * Mugi: Value Level Parallelism For Efficient LLMs \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790189)]
      * CMU & UCF
      * Introduce value-level parallelism as a new execution dimension for LLM inference.
      * Exploit finer-grained parallel structure than conventional tensor or sequence parallelism.
  * KV Cache Offloading
    * REPA: Reconfigurable PIM for the Joint Acceleration of KV Cache Offloading and Processing \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790212)]
      * SJTU
      * Present a reconfigurable PIM architecture for jointly offloading and processing KV cache.
      * Co-design KV movement and KV computation to reduce host-memory bottlenecks during inference.
    * STARC: Selective Token Access with Remapping and Clustering for Efficient LLM Decoding on PIM Systems \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790226)]
      * RPI & UMass Amherst & IBM Research
      * Introduce selective token access with remapping and clustering for PIM-based LLM decoding.
      * Reduce unnecessary KV accesses and improve data locality during decoding.
* LLM Training
  * RL Post-Training
    * Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790231)] \[[arXiv](https://arxiv.org/abs/2511.16665)] \[[Code](https://github.com/mit-han-lab/fastrl)]
      * MIT & NVIDIA & ETH Zurich & MIT-IBM AI Lab & UMass Amherst
      * Present **TLT**, a reasoning RL training system that accelerates long-tail rollout generation with adaptive speculative decoding.
      * Train an Adaptive Drafter on idle GPUs during long-tail generation and use an Adaptive Rollout Engine to select speculative-decoding strategies per input batch.
    * History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790172)]
      * SJTU & ByteDance
      * Present **RhymeRL**, a framework that accelerates RL by exploiting reusable structure across rollout histories.
      * Reduce redundant rollout work to improve training efficiency for LLM-aligned RL workloads.
  * Multimodal Model Training
    * DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790154)]
      * SJTU & StepFun & Zenergize AI
      * Introduce a dynamic interleaved pipeline for large multimodal model training.
      * Increase pipeline utilization by interleaving stages dynamically across modalities and training phases.
  * Mixed-Precision Training
    * SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790223)] \[[arXiv](https://arxiv.org/abs/2602.01410)]
      * UMich & Meta & UMass Amherst
      * Present an adaptive mixed-precision framework for subbyte LLM training.
      * Periodically profile training statistics and solve precision allocation to assign fine-grained bitwidths.
  * Diagnosis
    * Fine-grained and Non-intrusive LLM Training Monitoring via Microsecond-level Traffic Measurement \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790163)]
      * NJU & NUS & Infrawaves
      * Introduce microsecond-level traffic measurement for fine-grained, non-intrusive LLM training monitoring.
      * Infer communication and runtime behavior without intrusive application instrumentation.
  * Offloading
    * SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762217)] \[[arXiv](https://arxiv.org/abs/2509.21271)]
      * UIUC & Microsoft & Snowflake
      * Revisit large-scale LLM training on tightly coupled GPU-CPU superchips with **SuperOffload**.
      * Combine adaptive weight offloading with superchip-aware runtime optimizations to improve long-context training throughput.
* Language Processing Units (LPUs)
  * Hardwired-Neuron Language Processing Units as General-Purpose Cognitive Substrates \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790169)]
    * ICT, CAS & USTC & IS, CAS & Cambricon Technologies
    * Propose Language Processing Units (LPUs) as a language-centric hardware substrate for general-purpose cognitive workloads.
    * Specialize the architecture around language processing primitives to improve efficiency on language-centric tasks.

### Generative Recommenders (GRs)

* GR Serving
  * BAT: Efficient Generative Recommender Serving with Bipartite Attention \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790131)]
    * ZJU & HKU & Alibaba & NUS & Aalto University
    * Introduce bipartite attention for generative recommender serving.
    * Tailor the serving design to recommendation-style generative workloads rather than generic LLM inference.

### Diffusion Models

* Video DiT Training
  * DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762216)] \[[arXiv](https://arxiv.org/abs/2502.07590)]
    * CUHK & StepFun
    * Exploit dynamic sparsity to accelerate large-scale video DiT training.
    * Use hybrid sparsity-aware context parallelism to rebalance workloads under heterogeneous attention sparsity.
* Diffusion Model Serving
  * TetriServe: Efficiently Serving Mixed DiT Workloads \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790233)] \[[arXiv](https://arxiv.org/abs/2602.05116)] \[[Code](https://github.com/DiT-Serving/TetriServe)]
    * UMich & UW-Madison & NTU
    * Present a serving system for mixed DiT workloads.
    * Coordinate scheduling and batching across heterogeneous diffusion requests in a shared runtime.
* Mixture-of-Diffusion Models
  * MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762220)] \[[Code](https://github.com/stsxxx/MoDM)]
    * UMich & Intel Labs
    * Introduce mixture-of-diffusion models for image generation serving.
    * Use specialization across diffusion sub-models to improve efficiency and quality-cost tradeoffs.

### Deep Learning Training

* T-Control: An Efficient Dynamic Tensor Rematerialization System for DNN Training \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790230)]
  * ICT, CAS
  * Present a dynamic tensor rematerialization system for DNN training.
  * Adjust rematerialization online to balance memory savings and recomputation overhead.
* NotebookOS: A Replicated Notebook Platform for Interactive Training with On-Demand GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762230)]
  * GMU & Adobe Research & UVA
  * Present a replicated notebook platform for interactive model training with on-demand GPUs.
  * Combine notebook-centric workflow support with elastic GPU provisioning.

### Deep Learning Compilation

* FuseFlow: A Fusion-Centric Compilation Framework for Sparse Deep Learning on Streaming Dataflow \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790165)]
  * Stanford & SambaNova Systems & Barcelona Supercomputing Center
  * Present a fusion-centric compilation framework for sparse deep learning on streaming dataflow hardware.
  * Expand fusion opportunities for sparse operators to improve accelerator execution efficiency.
* Trinity: Three-Dimensional Tensor Program Optimization via Tile-level Equality Saturation \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790240)]
  * KAIST & FuriosaAI
  * Introduce tile-level equality saturation for three-dimensional tensor program optimization.
  * Use equivalence-based search to discover better accelerator-friendly tensor rewrites.
* RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790209)]
  * Alibaba Cloud
  * Present an automatic operator fusion framework for cascaded reductions on AI accelerators.
  * Target reduction-heavy operator chains that are poorly handled by existing compiler passes.
* Linear Layouts: Robust Code Generation of Efficient Tensor Computation Using F2 \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762221)]
  * GMU & OpenAI
  * Introduce linear-layout abstractions for tensor code generation.
  * Improve portability and performance by reducing reliance on brittle layout-specific code generation.

### GPU Systems

* GPU Scheduling
  * gShare: Efficient GPU Sharing with Aggressive Scheduling in Multi-tenant FaaS Platform \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790168)]
    * China Telecom Cloud Computing Research Institute & China Telecom Cloud Technology Co. Ltd.
    * Present an aggressive GPU sharing and scheduling framework for multi-tenant FaaS platforms.
    * Improve utilization through fine-grained temporal multiplexing across serverless tenants.
  * GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762231)]
    * SJTU & ZJU & Alibaba
    * Present a preemption-aware scheduling framework for GPU clusters with predictive spot instance management.
    * Jointly schedule jobs and volatile spot capacity to reduce disruption and improve cluster efficiency.
* GPU Communication
  * MSCCL++: Rethinking GPU Communication Abstractions for AI Inference \[[Paper](https://dl.acm.org/doi/10.1145/3779212.3790188)] \[[Code](https://github.com/microsoft/mscclpp)]
    * MSR & Microsoft Azure
    * Present a new GPU communication abstraction stack tailored to AI inference.
    * Move beyond training-centric collective abstractions to better support inference communication patterns.
* GPU Programming
  * cuJSON: A Highly Parallel JSON Parser for GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762222)]
    * UC Riverside
    * Introduce a highly parallel JSON parser for GPUs.
    * Make JSON parsing a scalable GPU primitive for preprocessing and data-serving pipelines.
  * CHERI-SIMT: Implementing Capability Memory Protection in GPGPUs \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762234)]
    * Cambridge
    * Implement capability-based memory protection in GPGPUs with CHERI-SIMT.
    * Bring stronger spatial memory safety and isolation to SIMT execution.
  * Lobster: A GPU-Accelerated Framework for Neurosymbolic Programming \[[Paper](https://dl.acm.org/doi/10.1145/3760250.3762232)]
    * UPenn
    * Present a GPU-accelerated framework for neurosymbolic programming.
    * Provide systems support for workloads that combine symbolic and neural computation.

### Profiling

* Deep Learning Profiling
  * DeepContext: A Context-aware, Cross-platform, and Cross-framework Tool for Performance Profiling and Analysis of Deep Learning Workloads \[[Paper](https://dl.acm.org/doi/10.1145/3676642.3736127)]
    * NCSU & GMU
    * Introduce a context-aware profiling and analysis tool for deep learning workloads across platforms and frameworks.
    * Use execution context to explain performance behavior beyond isolated kernel-level statistics.

## Acronyms

* DiT: Diffusion Transformer
* GR: Generative Recommender
* LLM: Large Language Model
* LPU: Language Processing Unit
* MoE: Mixture-of-Experts
* PIM: Processing-in-Memory
* RL: Reinforcement Learning


# FAST 2026

## Meta Info

Homepage: <https://www.usenix.org/conference/fast26>

Paper list: <https://www.usenix.org/conference/fast26/technical-sessions>

### Acceptance Rate

17.5% (= 31 / 177)

## Papers

### AI for Systems

* Generative File Systems
  * Sharpen the Spec, Cut the Code: A Case for Generative File System with SYSSPEC \[[Personal Notes](/reading-notes/conference/fast-2026/sysspec)] \[[Paper](https://www.usenix.org/conference/fast26/presentation/liu-qingyuan)] \[[PDF](https://www.usenix.org/system/files/fast26-liu-qingyuan.pdf)] \[[arXiv](https://arxiv.org/abs/2512.13047)] \[[Code](https://github.com/SJTU-IPADS/specfs)] \[[Homepage](https://llmnativeos.github.io/specfs/)] \[[Slides](https://www.usenix.org/system/files/fast26_slides_liu-qingyuan.pdf)] \[[Video](https://www.youtube.com/watch?v=aMxAPuYhAkk)]
    * SJTU
    * **Best Paper**; **Distinguished Artifact Award**
    * Introduce generative file systems, where LLMs generate and evolve a file system from specifications instead of ambiguous natural-language prompts.
    * Use a multi-part SYSSPEC specification for functionality, modularity, and concurrency, plus DAG-structured patches for feature evolution without violating existing invariants.
    * Generate SPECFS, a concurrent file system that matches a manually coded baseline across regression tests and integrates 10 real-world Ext4 features.


# Sharpen the Spec, Cut the Code: A Case for Generative File System with SYSSPEC

\#generative\_file\_system #llm\_for\_systems #specification

## Meta Info

Presented in [FAST 2026](https://www.usenix.org/conference/fast26/presentation/liu-qingyuan).

Authors: Qingyuan Liu, Mo Zou, Hengbin Zhang, Dong Du, Yubin Xia, Haibo Chen (*SJTU*)

Awards: **Best Paper**; **Distinguished Artifact Award**

Resources: [Paper](https://www.usenix.org/conference/fast26/presentation/liu-qingyuan), [PDF](https://www.usenix.org/system/files/fast26-liu-qingyuan.pdf), [arXiv](https://arxiv.org/abs/2512.13047), [Code](https://github.com/SJTU-IPADS/specfs), [Homepage](https://llmnativeos.github.io/specfs/), [Slides](https://www.usenix.org/system/files/fast26_slides_liu-qingyuan.pdf), [Video](https://www.youtube.com/watch?v=aMxAPuYhAkk)

## Understanding the paper

### TL;DR

**SYSSPEC** makes the specification—not generated code—the source of truth for a file system. Developers describe behavior, module contracts, and concurrency; LLM agents generate and evolve the C implementation. Its main insight is that structured, composable specifications make whole-system generation more reliable than natural-language prompts or source-code context alone.

The motivation comes from Ext4's history: 82.4% of 3,157 commits are bug fixes or maintenance, while only 5.1% add features. File-system development is therefore mainly an evolution problem: many small changes must preserve invariants across tightly coupled modules.

### Key design

<figure><img src="https://819228986-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkzeiawY8SkBarQBDVm-659326392%2Fuploads%2Fgit-blob-8956ffc2dee3ed6ca9e41b8e1bedbe6a92d390b2%2FSysSpec-overview.png?alt=media" alt=""><figcaption><p>SYSSPEC separates system design, implementation generation and validation, and specification-level evolution.</p></figcaption></figure>

SYSSPEC separates three concerns:

* **Functionality** uses pre-/post-conditions, invariants, and optional algorithm or intent descriptions. Contracts state exactly which transitions are valid; algorithms constrain performance-sensitive choices that functional correctness alone cannot distinguish.
* **Modularity** keeps each module within the model's context window and connects modules through `Rely`/`Guarantee` contracts. Generation sees required interfaces and behaviors without loading the entire codebase.
* **Concurrency** specifies lock ownership and ordering separately. SpecCompiler first generates sequential logic, then instruments synchronization in a second pass, reducing a difficult synthesis problem into two narrower ones.

Evolution uses **DAG-structured spec patches**. Leaf nodes introduce local changes and new guarantees; intermediate nodes regenerate affected dependents; root nodes preserve the old external guarantee and become safe replacement points. The DAG records both change impact and generation order instead of asking an agent to rediscover dependencies from code.

SpecCompiler generates modules and retries with feedback from a separate SpecEval role. SpecValidator combines specification review with compilation and regression tests. SpecAssistant helps refine draft specifications. These agents mitigate hallucination; they do not prove correctness.

### Evidence

**SPECFS** is a 45-module, approximately 4.3-KLoC concurrent in-memory FUSE file system based on AtomFS's design. Gemini 2.5 Pro and DeepSeek V3.1 Reasoning generate all 45 evaluated modules correctly; an oracle baseline that sees dependency implementations reaches 81.8% with Gemini.

The ablation is the clearest evidence for the design. Functionality specifications alone generate 12/40 concurrency-agnostic modules. Adding modularity reaches 40/40. For five thread-safe modules, functionality plus modularity reaches 0/5; concurrency specifications raise this to 4/5, and SpecValidator closes the gap to 5/5. The authors also evolve SPECFS with 10 Ext4-derived features.

### Caveats

* SYSSPEC is **formal-method-inspired, not formally verified**: structured natural language, LLM review, and tests replace a theorem prover.
* The specification becomes the trusted artifact. Missing invariants or dependency changes can produce consistently wrong code.
* SPECFS is a userspace, in-memory prototype without direct disk access or crash consistency. The work demonstrates specification-driven generation and evolution, not a production replacement for Ext4.


# HPCA 2026

## Meta Info

Homepage: <https://conf.researchr.org/home/hpca-2026>

Paper list: <https://2026.hpca-conf.org/track/hpca-2026-main-conference#event-overview>

## Papers

### LLM

* LLM training
  * AutoHAAP: Automated Heterogeneity-Aware Asymmetric Partitioning for LLM Training
    * Zhejiang Lab
* LLM inference
  * AUM: Unleashing the Efficiency Potential of Shared Processors with Accelerator Units for LLM Serving \[[Paper](https://www.cs.sjtu.edu.cn/~lichao/publications/AUM_Unleashing_HPCA-2026-Wang.pdf)]
    * SJTU & Alibaba
  * GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference
    * KAIST
  * ELORA: Efficient LoRA and KV Cache Management for Multi-LoRA LLM Serving
    * SJTU & Huawei Cloud & HKUST
  * Towards Resource-Efficient Serverless LLM Inference with SLINFER \[[arXiv](https://arxiv.org/abs/2507.00507)]
    * SJTU
  * LILo: Harnessing the On-chip Accelerators in Intel CPUs for Compressed LLM Inference Acceleration
    * UIUC & Seoul National University & Intel
  * PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System \[[arXiv](https://arxiv.org/abs/2412.20166)]
    * Hanyang University & SK hynix & KAIST
* Speculative decoding
  * Adaptive Draft Sequence Length: Enhancing Speculative Decoding Throughput on PIM-Enabled Systems
    * HUST
* Wafer
  * WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chip \[[arXiv](https://arxiv.org/abs/2512.12279)]
    * THU
  * TEMP: A Memory Efficient Physical-aware Tensor Partition-Mapping Framework on Wafer-scale Chips \[[arXiv](https://arxiv.org/abs/2512.14256)]
    * THU
  * FACE: Fully PD Overlapped Scheduling and Multi-Level Architecture Co-Exploration on Wafer
    * THU
  * MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference \[[arXiv](https://arxiv.org/abs/2510.25258)]
    * THU
  * HDPAT: Hierarchical Distributed Page Address Translation for Wafer-Scale GPUs
    * William\&Mary
  * ReThermal: Co-Design of Thermal-Aware Static and Dynamic Scheduling for LLM Training on Liquid-Cooled Wafer-Scale Chips
    * THU & Shanghai AI Lab
* Quantization
  * BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache \[[arXiv](https://arxiv.org/abs/2503.18773)]
    * Edinburgh & MSRA
  * AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization
    * Institute of Science Tokyo
* Reasoning
  * The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective \[[arXiv](https://arxiv.org/abs/2506.04301)]
    * KAIST
  * PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models
    * KAIST
  * RPU - A Reasoning Processing Unit
    * Harvard
* RAG
  * VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG \[[arXiv](https://arxiv.org/abs/2504.08930)]
    * GaTech
* VLM
  * Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models \[[arXiv](https://arxiv.org/abs/2512.14661)] \[[Code](https://github.com/dubcyfor3/Focus)]
    * Duke
* Video LLM
  * V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval \[[arXiv](https://arxiv.org/abs/2512.12284)]
    * KAIST
* Misc
  * Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi-GPU Systems
    * SJTU & Huawei
  * RoMe: Row Granularity Access Memory System for Large Language Models \[[arXiv](https://arxiv.org/abs/2512.01541)]
    * Seoul National University & Meta
  * LEGO: Supporting LLM-enhanced Games with One Gaming GPU
    * SJTU & Tongji University

### GPU

* UVM
  * ARIADNE: Adaptive UVM Management for Efficient GPU Memory Oversubscription \[[Artifact](https://zenodo.org/records/17852674)]
    * Yonsei University & DGIST
* Chiplet
  * COMET: Communication and Memory Co-Design for Fine-Grained AI Inference in MCM Accelerators
    * NUDT & PKU
  * Deadlock-Free Bridge Module for Inter-Chiplet Communication in Open Chiplet Ecosystem
    * NUDT
  * LRM-GPU: Alleviating Synchronization Overhead for Multi-Chiplet GPU Architecture
    * SYSU
* Sparsity
  * Swift: High-Performance Sparse-Dense Matrix Multiplication on GPUs
    * Hunan University
  * Uni-STC: Unified Sparse Tensor Core
    * CUP-Beijing & NUDT
* Misc
  * QuCo: Efficient and Flexible Hardware-Driven Automatic Configuration of Tile Transfers in GPUs
    * University of Murcia & William\&Mary & NVIDIA
  * μShare: Non-Intrusive Kernel Co-Locating on NVIDIA GPUs
    * TJU
  * FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive operators via Inter-Core Connection \[[arXiv](https://arxiv.org/abs/2512.12949)]
    * SJTU

### VAR

* VAR-Turbo: Unlocking the Potential of Visual Autoregressive Models through Dual Redundancy
  * HKUST

## Acronyms

* LLM: Large Language Model
* VLM: Vision-Language Model
* RAG: Retrieval-Augmented Generation
* UVM: Unified Virtual Memory
* VAR: Visual AutoRegressive modeling


# PPoPP 2026

## Meta Info

Homepage: <https://ppopp26.sigplan.org/>

Paper list: <https://ppopp26.sigplan.org/track/PPoPP-2026-papers>

## Papers

### LLM

* LLM training
  * CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training
    * ICT, CAS & Ant Group
  * COCCL: A Collective Communication Library Supporting Easy Integration and Configuration of Customized Compression for Scalable LLM Training
    * ICT, CAS & CUHK-SZ
  * Elastor: Elastic and Efficient Model Partitioning and Checkpointing for Fault-tolerant Distributed Training
    * PKU
  * HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism \[[arXiv](https://arxiv.org/abs/2507.00394)] \[[Code](https://github.com/zxgx/Megatron-LM/tree/dev)]
    * NUS
* LLM inference
  * JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-context Inference
    * WHU
  * Laser: Unlocking Layer-Level Scheduling for Efficient Multi-SLO LLM Serving
    * SYSU
  * High-Throughput Non-Uniformly Quantized 3-bit LLM Inference
    * CUHK & HKUST
  * Accelerating Sparse Transformer Inference on GPU
    * CUP-Beijing & BUAA
* Attention
  * FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism
    * USTC & ICT, CAS
  * MetaAttention: A Unified and Performant Attention Framework Across Hardware Backends \[[arXiv](https://arxiv.org/abs/2502.15349)] \[[Code](https://github.com/microsoft/AttentionEngine)]
    * SJTU, IPADS & PKU & MSRA

### Diffusion Models

* Difflow: A Data-Characteristic-Aware Serving System for Diffusion Models
  * THU
* MixFusion: A Patch-Level Parallel Serving System for Mixed-Resolution Diffusion Models \[[Paper](https://doi.org/10.1145/3774934.3786420)] \[[arXiv](https://arxiv.org/abs/2501.09253)] \[[Code](https://github.com/desenSunUBW/mixfusion)]
  * UWaterloo & CMU & Rice
  * Use patch-level parallelism to batch mixed-resolution text-to-image diffusion requests.
  * Combine patch-tailored cache management with SLO-aware scheduling and online latency prediction.

### GNN

* APERTURE: Algorithm-System Co-Optimization for Temporal Graph Network Inference
  * BUAA
* ElasGNN: An Elastic Training Framework for Distributed GNN Training
  * BUAA
* TAC: Cache-based System for Accelerating Billion-Scale GNN Training on Multi-GPU Platform
  * UCAS

### Sparse Matrix

* ASM-SpMM: Unleashing the Potential of Arm SME for Sparse Matrix Multiplication Acceleration
  * SYSU
* Exploiting Efficient Mapping and Pipelined Execution for Accelerating SpMV on Tensor Cores
  * BUAA
* VDHA: Vector-Driven Hash Aggregation for Sparse Matrix–Sparse Vector Multiplication on GPUs
  * THU

### Quantization

* RoMeo: Mitigating Dual-dimensional Outliers with Rotated Mixed Precision Quantization \[[Artifact](https://github.com/thu-pacman/RoMeo)]
  * THU

### Cache Management

* Cacheman: A Comprehensive Last-Level Cache Management System for Multi-tenant Clouds
  * Alibaba Cloud

### Misc

* Scaling GPU-to-CPU Migration for Efficient Distributed Execution on CPU Clusters
  * GaTech
* zBuffer: Zero-Copy and Metadata-Free Serialization for Fast RPC with Scatter-Gather Reflection
  * XMU & Alibaba & SJTU

## Acronyms

* LLM: Large Language Model
* GNN: Graph Neural Network
* SpMM: Sparse Matrix-Matrix Multiplication
* SpMV: Sparse Matrix-Vector Multiplication
* RPC: Remote Procedure Call


# SC 2025

## Meta Info

Homepage: <https://sc25.supercomputing.org>

Paper list: <https://sc25.conference-program.com>

## Papers

### LLMs

* LLM Inference
  * Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism \[[Paper](https://doi.org/10.1145/3712285.3759784)] \[[arXiv](https://arxiv.org/abs/2509.08309)]
    * University of Macau & SYSU

## Acronyms

* LLM: Large Language Model


# SOSP 2025

## Meta Info

Homepage: <https://sigops.org/s/conferences/sosp/2025/>

### Paper List

* <https://sigops.org/s/conferences/sosp/2025/accepted.html>
* <https://dl.acm.org/doi/proceedings/10.1145/3731569>

### Acceptance Rate

17.7% (= 65 / 368)

## Papers

### LLM

* LLM Training
  * Robust LLM Training Infrastructure at ByteDance \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764838)]
    * HKU & ByteDance Seed
    * **ByteRobust**
  * Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764848)]
    * CUHK & ByteDance & ByteDance Seed
    * A lightweight distributed tracing and root cause analysis system.
    * Trace collective communication states and leverage internal control and data dependencies.
  * DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764849)]
    * HKU & AWS
    * Introduce fine-grained blockwise partitioning of both data and computation.
  * TrainVerify: Equivalence-Based Verification for Distributed LLM Training \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764850)]
    * UMich & MSRA
    * Formally verify that a distributed parallel execution plan is mathematically equivalent to the logical specification.
    * Introduce a stage-wise parallel verification algorithm and shape-reduction techniques → Reduce complexity while preserving formal correctness.
* LLM Inference
  * Jenga: Effective Memory Management for Serving LLM with Heterogeneity \[[Personal Notes](/reading-notes/conference/sosp-2025/jenga)] \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764823)] \[[arXiv](https://arxiv.org/abs/2503.18292)]
    * THU & UChicago & UC Berkeley
    * Two challenges
      * Recent models have heterogeneous embeddings with different sizes.
      * Some new architectures use only a subset of the prefix tokens to generate the next token.
    * Designs
      * Two-level memory allocator: choose the page size as least common multiple of token embedding sizes.
      * Enable attention variants to customize this mechanism by precisely specifying the exact prefix subset.
  * PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764834)] \[[arXiv](https://arxiv.org/abs/2505.07203)]
    * Chicago & THU & LinkedIn & UC Berkeley
    * Hybrid prefilling: Prefill non-attention layers chunk-by-chunk, but prefill the attention layers normally.
    * Suffix KV cache discarding / offloading: Discard the useless KV cache.
    * Continuous JCT calibration: Continuously reestimate the JCT of each request based on what requests are previously scheduled, and then schedule just one request with the lowest JCT.
  * IC-Cache: Efficient Large Language Model Serving via In-context Caching \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764829)]
    * UIUC & Google
    * Leverage historical request-response pairs from larger models as in-context examples.
  * Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764815)]
    * PKU & Alibaba Cloud
    * Schedule multimodel requests and make auto-scaling decisions on a per-token basis to maximize service quality.
    * Reduce auto-scaling overhead through component reuse, explicit memory management, and fine-grained KV cache synchronization.
  * Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764808)]
    * SJTU IPADS & THU & SenseTime
* LLM Applications
  * Pie: A Programmable Serving System for Emerging LLM Applications \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764814)]
    * Yale
    * Decompose the traditional generation loop into fine-grained service handlers exposed via an API and delegates control of the generation process to user-provided programs, called *inferlets*.
    * Enable applications to implement new KV cache strategies, bespoke generation logic, and seamlessly integrate computation and I/O—entirely within the application, without requiring modifications to the serving system.
* RAG Systems
  * METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764855)] \[[arXiv](https://arxiv.org/abs/2412.10543)]
    * Chicago & Princeton & MSR
    * Jointly schedule queries and adapt the key RAG configurations of each query (e.g., the number of retrieved text chunks, synthesis methods).
  * HeteRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764806)] \[[arXiv](https://arxiv.org/abs/2507.09138)] \[[Artifact](https://github.com/Leo9660/HedraRAG_AE)]
    * UCSD
    * RAGraph, a graph-based abstraction → Expose optimization opportunities across stage-level parallelism, intra-request similarity, and inter-request skewness.
* KV Cache Management
  * DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764810)]
    * Huawei & CUHK & SJTU
    * Exploit three levels of differentiation in the KV cache:
      * The differing impact of keys and values on attention computation.
      * The varying importance of tokens.
      * The diverse dynamic sparsity patterns across attention heads.
    * An on-GPU memory manager → Compact fragmented free memory list into contiguous regions in parallel.
* Multi-GPU Operator Optimization
  * Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764798)] \[[Artifact](https://github.com/ChandlerGuan/mercury_artifact)]
    * UCSD & Meta
    * A multi-GPU operator compiler based on a loop-based intermediate representation, CommIR.
    * Treat remote GPU memory as an explicitly managed extension of the memory hierarchy.
    * Automatically reproduce the performance of hand-optimized baselines like RingAttention and Ulysses.

### MoE

* MoE Inference
  * KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764843)] \[[Code](https://github.com/kvcache-ai/ktransformers)]
    * THU & Approaching.AI
    * Employ optimized, AMX-specialized kernels to fully utilize the computational capabilities of modern CPUs and incorporate an asynchronous CPU-GPU task scheduling mechanism to minimize overhead.
    * Expert Deferral → Overlap CPU and GPU computations.

### Distributed Training

* Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764839)]
  * ETH & MIT & HES-SO
  * Combine an efficient search space exploration algorithm, accurate runtime and memory footprint simulation, and a distributed training framework → Support different types of heterogeneity to optimize training throughput and cost.

### Deep Learning Compilation

* Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764840)]
  * ICL
  * Temporal relationships: a tensor at one timestep may depend on tensors from earlier or later timesteps.
  * Construct a symbolic dependence graph → Concisely encode dynamic dependencies between operators, and apply whole-program optimizations.

### GPU

* GPU OS
  * LithOS: An Operating System for Efficient Machine Learning on GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764818)] \[[arXiv](https://arxiv.org/abs/2504.15465)]
    * CMU & Meta
    * A TPC Scheduler → Support spatial scheduling at the granularity of individual TPCs.
    * A kernel atomizer → Reduce head-of-line blocking and allow dynamic resource reallocation mid-execution.
    * A lightweight hardware right-sizing mechanism → Dynamically determine the minimal TPC resources needed per atom.
    * A power management mechanism → Reduce power consumption based upon in-flight work characteristics.
    * Built in Rust.
* GPU Checkpointing
  * PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764813)] \[[arXiv](https://arxiv.org/abs/2405.12079)]
    * SJTU IPADS
    * Proactively detect GPU memory reads and writes through a two-step process:
      * Speculate about GPU memory accesses based on the arguments used when launching GPU kernels.
      * Validate these accesses efficiently at runtime using binary instrumentation.
    * Coordinated checkpoint data transfer and execution context pool.
* GPU Storage
  * Managing Scalable Direct Storage Accesses for GPUs with GoFS \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764857)]
    * UIUC
    * GPU-orchestrated file system to offload the storage management to the GPU → Scale the direct storage accesses for GPU programs.

### RDMA

* Live Migration
  * Device-Assisted Live Migration of RDMA Devices \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764795)]
    * NVIDIA
    * A generic device-hypervisor interface.
    * The design and implementation of live migration support for the NVIDIA ConnectX family of network adapters.
    * Quiesce direct communication over the memory fabric (e.g., PCIe).

### CXL

* PCIe Pooling
  * Oasis: Pooling PCIe Devices Over CXL to Boost Utilization \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764812)]
    * Columbia & Microsoft Azure
    * Provide a control plane and datapath over CXL pools → Map and route PCIe device traffic across host boundaries.

### OS

* Proto: A Guided Journey through Modern OS Construction \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764811)]
  * UVA
* How to Copy Memory? Coordinated Asynchronous Copy as a First-Class OS Service \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764800)]
  * SJTU IPADS & Huawei
  * Copier, a new OS service of *coordinated asynchronous copy*, to serve both user-mode applications and OS services.

### Resource Management

* Serverless Computing
  * Unlocking True Elasticity for the Cloud-Native Era with Dandelion \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764803)]
    * ETH
    * Dandelion, an elastic cloud platform with *a declarative cloud-native programming model* that replaces POSIX-based network interfaces with higher-level (e.g., HTTP-based) interfaces for applications to interact with remote services (e.g., cloud storage, databases, and AI inference services).
    * Execute applications expressed as DAGs of pure compute functions and communication functions.
  * Quilt: Resource-aware Merging of Serverless Workflows \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764830)]
    * UPenn
    * Automatically merge workflows that consist of many functions (possibly in different languages) into one process → Avoid high invocation latency, communication overhead, and long chains of cold starts.
* Resource Allocation
  * COpter: Efficient Large-Scale Resource-Allocation via Continual Optimization \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764846)]
    * Microsoft & Meta & CMU
    * Reframe round-based resource allocation as a sequence of interconnected problems.
    * Provide a method for continual optimization of LP and MILP formulations of resource allocation problems.
* Cloud Deployment
  * Moirai: Optimizing Placement of Data and Compute in Hybrid Clouds \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764802)]
    * CMU & Uber

### Video

* SAND: A New Programming Abstraction for Video-based Deep Learning \[[Paper](https://dl.acm.org/doi/10.1145/3731569.3764847)]
  * KAIST
  * Integrate system-level optimizations to simplify the preprocessing pipeline and maximize resource efficiency.

## Acronyms

* OS: Operating System
* LLM: Large Language Model
* MoE: Mixture-of-Experts
* RAG: Retrieval Augmented Generation
* RDMA: Remote Direct Memory Access
* CXL: Compute Express Link
* LP: Linear Program
* MILP: Mixed Integer Linear Program


# Jenga: Effective Memory Management for Serving LLM with Heterogeneity

## Meta Info

Presented in [SOSP 2025](https://dl.acm.org/doi/10.1145/3731569.3764823).

Authors: Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, Jidong Zhai, Joseph Gonzalez, Ion Stoica (*THU, UChicago, UC Berkeley*)

## Understanding the paper

### TL;DR

Jenga is a KV-cache memory manager for LLM serving under heterogeneous model architectures. The core idea is to generalize PagedAttention's fixed-page abstraction into an attention-property-aware allocator: an LCM-sized compatible page layer for sharing memory across heterogeneous cache types, and per-type small-page allocators plus evictors for self-attention, sliding-window attention, Mamba states, vision embeddings, and multi-model/speculative-decoding caches.

Jenga's key contribution is not only smaller memory fragmentation. It also gives each layer type a way to define its own cache-hit and eviction semantics, while a global prefix-subset evictor keeps evictions balanced and aligned across layer types. This lets serving engines reclaim memory that fixed full-prefix KV-cache assumptions would keep unnecessarily.

### Background

* LLM serving throughput is often constrained by GPU memory, because larger batches require storing KV caches for all active requests.
* PagedAttention reduces fragmentation by mapping logical KV-cache pages to physical pages, but it assumes:
  * Fixed-size embeddings: every token/layer cache object fits the same page-size abstraction.
  * Full-prefix dependency: the next token depends on the whole prefix, so all prefix tokens share the same lifecycle.
* Modern LLM architectures break both assumptions:
  * Sparse/sliding-window attention only needs a recent prefix subset in some layers.
  * Mamba/state-space layers maintain large recurrent states instead of per-token KV for every prefix token.
  * VLMs combine text-token KV cache, image-token KV cache, and vision embedding cache with different sizes and lifecycles.
  * Speculative decoding and multi-model serving maintain caches for draft and target models with different sizes.

### Key observations

* Heterogeneous cache sizes cause memory fragmentation.
  * A fixed page size that matches one cache type can waste memory for another cache type.
  * For VLMs, allocating both image-token and text-token KV cache for all layers wastes memory because self-attention layers need text-token KV while cross-attention layers need image-token KV.
* Heterogeneous token dependencies change prefix caching.
  * Self-attention requires all prefix tokens to remain cached for a prefix hit.
  * Sliding-window attention only requires the tokens inside the current window.
  * Mamba layers may only cache sparse recurrent states, such as every fixed interval of tokens.
  * Vision embedding caches should prefer evicting whole images instead of scattering evictions across many images, because one missing image token can force recomputing the vision encoder.
* Eviction across layer types must satisfy two properties:
  * Balanced eviction: no layer type should evict so aggressively that it becomes the bottleneck for model-wide prefix hits.
  * Aligned eviction: different layer types should evict similar token sets, otherwise the union of evicted tokens destroys cross-layer prefix-cache hits.

### Design

<figure><img src="https://819228986-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkzeiawY8SkBarQBDVm-659326392%2Fuploads%2Fgit-blob-bd197b3e1f5c08f6f5dea5127c235e2a24df9cb2%2FJenga-overview.png?alt=media" alt=""><figcaption><p>Overview of Jenga's two-level memory manager.</p></figcaption></figure>

* Two-level memory allocation
  * Compatibility layer: partition the KV-cache memory into large pages whose size is the least common multiple (LCM) of all relevant cache object sizes.
  * Customization layer: each layer/cache type divides large pages into its own small pages, such as pages for self-attention KV, sliding-window KV, Mamba states, or vision embeddings.
  * This keeps a fixed exchange granularity across cache types while allowing each type to use an appropriate local page size.
* Request-aware allocation
  * Jenga tries to allocate small pages belonging to the same request from the same large page.
  * When a request finishes, those small pages tend to become free together, so the whole large page can be returned to the LCM allocator.
  * This reduces internal fragmentation inside large pages.
* Execution-compatible memory layout
  * Standard PagedAttention uses a layer-page layout.
  * Jenga switches to a page-layer layout so that each small page is contiguous and can be exchanged across cache types.
  * Existing PagedAttention kernels can still be reused by passing per-layer `kv_cache_start_ptr`, `page_size_exec`, and `pageid_exec`; Jenga does not require new CUDA attention kernels.
* Customizable prefix caching
  * `update_last_access` lets a layer type customize which pages should be considered recently used.
  * `set_prefix_length` lets a layer type align eviction priority across pages that represent the same logical prefix.
  * `get_possible_prefix` lets a layer type define which prefixes are valid cache hits given the current cached/missing-token bitmap.
* Prefix-subset evictor
  * A global evictor coordinates the customized per-layer evictors.
  * For cache hit, it asks every layer type for valid prefixes and chooses the longest common prefix that is valid for all layer types.
  * For eviction, it keeps last-access timestamps and prefix-length priorities coordinated so that different layer types evict compatible token sets.

### Layer-specific policies

* Sliding-window attention
  * Updates last-access time only for tokens inside the sliding window.
  * Allows prefix hits as long as the required recent window tokens remain cached.
  * Prioritizes evicting tokens outside the active window.
* Mamba layers
  * Avoids caching every token because Mamba states are large.
  * Caches sparse states, such as every 512 tokens in the prototype.
  * Exposes valid hit prefixes through `get_possible_prefix`.
* Vision embedding and cross-attention cache
  * Treats vision embedding cache as another heterogeneous cache type.
  * For chunked prefill, frees or reuses vision embeddings after they have been consumed by the LLM part.
  * Uses image-level eviction priority so recomputation tends to be concentrated on fewer images.
* Speculative decoding and multi-model serving
  * Handles draft-model and target-model KV caches as different cache types.
  * Uses the same compatible large-page layer to share memory across models with different KV-cache sizes.

### Implementation

* Implemented on top of vLLM.
* Around 4K lines of Python in the arXiv prototype.
* No CUDA kernel changes are required.
* The prototype parses model structures to discover possible embedding/cache sizes and configure memory allocation automatically.
* The arXiv prototype reports compatibility with all 90 models supported by vLLM v0.6.4.

### Evaluation

* Setup
  * Platforms: NVIDIA H100 80GB and NVIDIA L4 24GB.
  * Baseline: vLLM with only the memory-management subsystem changed.
  * Models include Llama 3.2 Vision, Gemma-2, Ministral, Jamba-1.5, a character.ai-style model, PyramidKV, and standard Llama.
  * Datasets include MMLU-pro, MMMU-pro, and long-context arXiv-QA workloads.
* End-to-end serving
  * The SOSP version reports up to 83% GPU memory-utilization improvement and up to 2.16x serving-throughput improvement, 1.46x on average.
  * Jenga preserves latency under low load, and improves latency under higher load because it can admit larger batches.
* Fragmentation breakdown
  * On Ministral traces, vLLM wastes a large fraction of KV-cache memory because it keeps sliding-window-layer KV for tokens outside the window.
  * Jenga reduces this waste by independently allocating self-attention and sliding-window KV cache and dynamically shifting memory between them.
* Prefix caching
  * For workloads with repeated articles and different questions, customized sliding-window eviction improves cache hit rate because old tokens outside the window can be evicted first.
  * Higher hit rate reduces recomputation and improves throughput.
* VLM chunked prefill
  * Jenga caches vision embeddings so the vision encoder does not need to rerun for each chunked-prefill step.
  * The arXiv prototype reports 1.88x throughput improvement and 1.60x latency improvement for this case.
* Speculative decoding
  * Jenga automatically manages different cache sizes for draft and target models.
  * It reaches the manually designed allocation performance for homogeneous Llama-style speculative decoding, and improves heterogeneous speculative decoding without model-specific allocator redesign.

### Limitations and discussion

* The LCM large-page size can become large when cache object sizes have unfavorable ratios. The paper argues this is manageable for models supported by vLLM v0.6.4, but this remains an engineering risk for future architectures.
* Jenga focuses on memory management inside the serving engine. It complements, rather than replaces, scheduling, offloading, KV compression, and distributed serving optimizations.
* Prefix-cache policy quality depends on exposing correct layer-specific dependency rules. New attention variants still need corresponding small policy implementations.
* Multi-model serving beyond speculative decoding is discussed as a natural extension, but full general support is left as future work.


# SIGCOMM 2025

## Meta Info

Homepage: <https://conferences.sigcomm.org/sigcomm/2025/>

### Paper List

* <https://conferences.sigcomm.org/sigcomm/2025/accepted-papers/>
* <https://dl.acm.org/doi/proceedings/10.1145/3718958>

### Acceptance Rate

16% (= 74 / 460 (approx.))

## Papers

### Large Language Models (LLMs)

* Infrastructure
  * InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750468)] \[[Video](https://www.youtube.com/watch?v=d4PX36vVDX0)]
    * PKU & StepFun & Lightelligence
    * Key insight: unify connectivity and dynamic switching at the transceiver level using OCS.
    * Realize the transceiver-centric HBD architecture in production → Flexible construction of arbitrarily large ring topologies & improved system resilience
      * Silicon Photonics (SiPh) based OCS transceiver (OCSTrx)
      * Reconfigurable k-hop ring topology → Each node connects to all other nodes within ≤𝐾 hops via OCSTrx
      * HBD-DCN orchestration algorithm → Minimize cross-ToR traffic
  * Astral: A Datacenter Infrastructure for Large Language Model Training at Scale \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750521)] \[[Video](https://www.youtube.com/watch?v=Ou389tkXL1I)]
    * NJU & Tencent
* LLM Training
  * DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750472)] \[[Video](https://www.youtube.com/watch?v=O_qdjBsLwE8)] \[[arXiv](https://arxiv.org/abs/2408.04275)]
    * PKU & StepFun
    * Disaggregated model orchestration: separate the training for modality encoder (ViT for images, Beats for audios), LLM backbone, and modality generator (Diffusion for images, AudioLDM for audio).
    * Disaggregated data preprocessing: decouple data preprocessing from training.
    * Integrated with Megatron-LM.
  * ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3754352)] \[[Video](https://www.youtube.com/watch?v=cXsMyc7ROyo)]
    * PKU & ByteDance
    * Limitations of existing works
      * The mismatch between data heterogeneity & static mesh → Redundant communication and imbalanced computation.
    * **HDP**: Hybrid Data Parallelism
      * Unify the inter- and intra-data partitioning with a dynamic mesh design.
      * A communication optimizer
        * Eliminate the redundant communication for *short sequences* by *data-aware sharding* and *dynamic communication*.
        * Compress the communication cost for *long sequences* by *selective offloading*.
      * A balance scheduler → Mitigate the imbalanced computation by *parallelism-aware data assignment*.
  * From ATOP to ZCube: Automated Topology Optimization Pipeline and A Highly Cost-Effective Network Topology for Large Model Training \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750503)] \[[Video](https://www.youtube.com/watch?v=7fVx3b1T_dc)]
    * THU & Zhongguancun Laboratory & Harnets.AI & ByteDance
    * ATOP: Automated Topology Optimization Pipeline
      * Model network topology as a set of hyperparameters → Enable the discovery of potential network topologies.
    * A new topology ZCube, discoverd by ATOP.
      * Reach the highest cost-effectiveness across various GPU scale.
  * SkeletonHunter: Diagnosing and Localizing Network Failures in Containerized Large Model Training \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750513)] \[[Video](https://www.youtube.com/watch?v=s1hPuZ033Nk)]
    * THU & Alibaba Cloud
    * Key idea: reason about the traffic skeleton, which comprises a crucial set of network paths consistently traversed by the training traffic.
* Privacy-preseving LLM Inference
  * SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer Serving \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750509)] \[[Video](https://www.youtube.com/watch?v=isix2nPNkyI)] \[[Code](https://github.com/yuanmu97/scx)]
    * CUHK
    * Encode the intermediate key-value cache using user-controlled keys → Ensure that the cloud can neither recover the input nor independently complete the next token prediction.
* LLMOps
  * Intent-Driven Network Management with Multi-Agent LLMs: The Confucius Framework \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750537)] \[[Video](https://www.youtube.com/watch?v=QpCXjseK0HQ)]
    * Meta
    * Model network management workflows as DAGs to aid planning.
    * Integrate LLMs with existing *management tools* to achieve seamless operational integration, employ RAG to improve long-term memory, and establish a set of primitives to systematically support *human/model interaction*.
    * Integrate with existing *network validation* methods and incorporate its own validation framework to prevent regressions.
  * Towards LLM-Based Failure Localization in Production-Scale Networks \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750505)] \[[Video](https://www.youtube.com/watch?v=95NQmZrTwbk)]
    * NJU & Alibaba Cloud
    * BiAn (狴犴), an LLM-based framework for efficient incident investigation.
    * Process monitoring data and generate error device rankings with detailed explanations.

### Mixture-of-Experts (MoEs)

* MoE Training
  * MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750465)] \[[Video](https://www.youtube.com/watch?v=zYVtGsWzAjI)]
    * HKUST
    * Design and implement a regionally reconfigurable HBD that augments existing electrical interconnects using OCS.
* MoE Inference
  * MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750506)] \[[Video](https://www.youtube.com/watch?v=eXHC8OpI2hk)] \[[arXiv](https://arxiv.org/abs/2504.02263)]
    * PKU & ByteDance
    * Attention/FFN Disaggregation (AFD)
    * Provide a M2N communication library → Eliminate unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization.

### RDMA

* Reliability
  * Revisiting RDMA Reliability for Lossy Fabrics \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750480)] \[[Video](https://www.youtube.com/watch?v=4CgsjKMJ1Ns)]
    * HKUST & Huawei
    * **Best Student Paper Award (Honorable Mention)**
    * DCP co-designs the switch and RNICs, including DCP-Switch and DCP-RNIC.
      * Header-only-based retransmission.
      * Bitmap-free packet tracking.
    * Prototype DCP-Switch using P4 switch and DCP-RNIC using FPGA.
* Virtualization
  * Software-based Live Migration for RDMA \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750487)] \[[Video](https://www.youtube.com/watch?v=_JTZhc6wYxo)]
    * THU & MSRA
    * MigrRDMA: a software-based RDMA live migration.
    * Provide a software indirection layer to achieve transparent switching to new RDMA communications.
    * Implemented over Mellanox RNICs.
  * ByteDance Jakiro: Enabling RDMA and TCP over Virtual Private Cloud \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750496)] \[[Video](https://www.youtube.com/watch?v=gh13ILZGY1s)]
    * ByteDance
    * Support fundamental VPC features (e.g., QoS, security groups) for both RDMA and TCP streams.
  * Alibaba Stellar: A New Generation RDMA Network for Cloud AI \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750539)] \[[Video](https://dl.acm.org/doi/10.1145/3718958.3750539)]
    * Alibaba Cloud
    * Limitations of existing RDMA virtualization solutions (e.g., SR-IOV)
      * Host-level
        * The number of VFs is static → Cannot dynamically scale the number of VFs.
        * The container must pin all of its memory in the host memory before initiating any RDMA operation → A minute-level start-up delay.
      * PCIe-level
        * LUT in PCIe fabrics is severely limited in size → Only a small number of VFs to enable GDR.
      * RNIC-level
        * No support for strict isolation between RDMA and non-RDMA traffic.
    * Three designs
      * Para-Virtualized Direct Memory Access (**PVDMA**) for on-demand memory pinning → Reduce host memory consumption & mitigate the start-up delay of secure containers.
      * Extended Memory Translation Table (**eMTT**) for optimized GDR performance → Allow the RNIC to bypass unnecessary consultations of memory address mappings in the PCIe fabric.
      * RDMA Packet Spray for efficient multi-path utilization
* Performance Diagnosis
  * Hawkeye: Diagnosing RDMA Network Performance Anomalies with PFC Provenance \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750490)] \[[Video](https://www.youtube.com/watch?v=wp5h3cFMXsk)]
    * THU & BUAA & Infrawaves
    * Three designs
      * A PFC-aware telemetry mechanism → Record the PFC impact on flows
      * An in-network PFC causality analysis and tracing mechanism → Collect causal telemetry for diagnosis
      * A provenance-based diagnosis algorithm → Present the anomaly breakdown, identify the anomaly type and root causes
    * Evaluated on both NS-3 simulations and a Tofino testbed.
* I/O Acceleration
  * CEIO: A Cache-Efficient Network I/O Architecture for NIC-CPU Data Paths \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750488)] \[[Video](https://www.youtube.com/watch?v=yMJG4m56-eo)] \[[Code](https://github.com/axio-project/ceio)]
    * HKUST
    * Limitations of traditional I/O acceleration strategies (e.g., Data Direct I/O (DDIO), RDMA)
      * Inefficient utilization of the LLC.
    * Cache-efficient I/O → Line-rate throughput and µs-scale tail latency
      * Limit I/O Rate → Proactive rate control
      * Limit I/O Capacity → Elastic buffer
    * Implemented on commodity SmartNICs and incorporated into DPDK and RDMA libraries.

### Hardware Transport

* Falcon: A Reliable, Low Latency Hardware Transport \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3754353)] \[[Video](https://www.youtube.com/watch?v=nu18DsuvnlU)]

  * Google
  * Support multiple Upper Layer Protocols (ULPs) and heterogeneous application workloads in general-purpose Ethernet datacenter environments (with losses and without special switch support).
  * Key designs: delay-based congestion control with multipath load balancing, a layered design with a simple request-response transaction interface for multi-ULP support, hardware-based retransmissions and error-handling for scalability, a programmable engine for flexibility.
  *

  ```
  <figure><img src="../../.gitbook/assets/Falcon-hardware-transport-layers.png" alt=""><figcaption><p>Falcon hardware transport layers</p></figcaption></figure>
  ```

### Collective Communication

* ResCCL: Resource-Efficient Scheduling for Collective Communication \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750514)] \[[Video](https://www.youtube.com/watch?v=uu594-CfWNE)]
  * NEU & SIAT, CAS & Alibaba Cloud
  * Limitations of existing works (e.g., NCCL, RCCL, MSCCL)
    * Static resource allocation and scheduling mechanisms → Inefficient utilization of bandwidth and SM resources for various collective algorithms
  * Three designs
    * Optimize scheduling at the primitive level (e.g., send and recvReduceCopy).
    * Enable flexible thread block allocation.
    * Generate lightweight communication kernels to minimize runtime overhead.
* SyCCL: Exploiting Symmetry for Efficient Collective Communication Scheduling \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750499)] \[[Video](https://www.youtube.com/watch?v=x1vL9SbcZmE)]
  * Alibaba Cloud & THU
  * Limitations of existing works
    * Existing collective communication libraries (e.g., NCCL, RCCL) rely on fixed schedules and cannot adjust to varying topology and model requirements.
    * Existing collective schedule synthesizers (e.g., TECCL, TACCL) utilize Mixed Integer Linear Program for modeling but encounter scalability challenges.
  * SyCCL, a scalable collective schedule synthesizer → Synthesize near-optimal schedules in tens of minute.
    * Leverage collective and topology symmetries to decompose the original collective communication demand into smaller sub-demands within smaller topology subsets.
    * Propose efficient search strategies to explore potential sub-demands, synthesizes corresponding sub-schedules, and integrates these sub-schedules into complete schedules.

### Video Streaming

* Towards User-level QoE: Large-scale Practice in Personalized Optimization of Adaptive Video Streaming \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750526)] \[[Video](https://www.youtube.com/watch?v=vWqnKwmlkwQ)]
  * THU & Kuaishou & SFU
  * **LingXi**, a system for personalized adaptive video streaming.
    * Dynamically optimize the objectives of adaptive video streaming algorithms by analyzing user engagement.
    * Iteratively determine optimal parameters through Monte Carlo sampling and online Bayesian optimization.
* TLadder: QoE-Centric Video Ladder Optimization with Playback Feedback at Billion Scale \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750500)] \[[Video](https://www.youtube.com/watch?v=pKPHFzR3Cww)]
  * ByteDance
  * Jointly consider the video content dimension (i.e., the bitrate-quality tradeoff of candidate representations) and the playback feedback dimension (e.g., network condition, rebuffering time, and playback bitrate).
* ACE: Sending Burstiness Control for High-Quality Real-time Communication \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750520)] \[[Video](https://www.youtube.com/watch?v=_c8fOxVZsAU)]
  * HKUST & ByteDance
  * A dual-control approach that manages both the encoding and transmission burstiness.
    * Sender: dynamically adjust the bucket size of a token-based pacer to control burstiness at the granularity of frame level.
    * Encoder: an adaptive complexity mechanism that smoothens frame sizes without sacrificing quality.
* Harnessing WebRTC for Large-Scale Live Streaming \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750535)] \[[Video](https://www.youtube.com/watch?v=Bf_zTLZ2C5w)]
  * ByteDance
  * Focus on optimizing first-frame delay, startup video rebuffering, audio-to-video drift, and per-session CPU usage.
* Scalable Video Conferencing Using SDN Principles \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750489)] \[[Video](https://www.youtube.com/watch?v=_8v9ZTrFkZI)] \[[Code](https://github.com/princeton-cabernet/scallop)]
  * Princeton & UVA
  * **Scallop**, an SDN-inspired SFU (Selective Forwarding Unit)
    * Decouple video-conferencing applications into a hardware-based data plane for latency-sensitive and frequent media operations.
  * A software control plane for the (infrequent) remaining tasks (e.g., analyze feedback signals, session management).

### CXL

* Understanding and Profiling CXL.mem Using PathFinder \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750479)] \[[Video](https://www.youtube.com/watch?v=PHqrdRGRfD4)] \[[Code](https://github.com/netlab-wisconsin/PathFinder)]
  * UW-Madison & BUAA & Intel
  * Leverage the capabilities of existing PMUs and dissect the `CXL.mem` protocol at adequate granularities.
  * Key idea: view the server processor and its chipset as a multi-stage Clos network, equip each architectural module with a PMU-based telemetry engine, track different `CXL.mem` paths, and apply conventional traffic analysis techniques.
  * Perform snapshot-based path-driven profiling and introduce four techniques: path construction, stall cycle breakdown, interference analyzer, and cross-snapshot analysis.
  * Built atop Linux Perf.

### Network Failures

* SkyNet: Analyzing Alert Flooding from Severe Network Failures in Large Cloud Infrastructures \[[Paper](https://dl.acm.org/doi/10.1145/3718958.3750536)] \[[Video](https://www.youtube.com/watch?v=3NhCnKqgBtM)]
  * Alibaba Cloud
  * Extract scope and severity information from alert floods.
  * Integrate multiple monitoring data sources through a uniform input format.

## Acronyms

* RDMA: Remote Direct Memory Access
* OCS: Optical Circuit Switching
* HBD: High-Bandwidth Domain
* DCN: Datacenter Network
* ToR: Top-of-Rack
* VPC: Virtual Private Cloud
* SR-IOV: Single-Root Input/Output Virtualization
* GDR: GPUDirect RDMA
* VF: Virtual Function
* LUT: Look-Up Table
* LLC: Last-Level Cache
* CXL: Compute Express Link
* PMU: Performance Monitoring Unit
* WebRTC: Web Real-Time Communications
* DAG: Directed Acyclic Graph
* RAG: Retrieval-Augmented Generation


# ICML 2025

## Meta Info

Homepage: <https://icml.cc/Conferences/2025>

### Acceptance Rate

* 26.9% (= 3260 / 12107)
* Poster: 24.3% (= 2947 / 12107)
* Spotlight: 2.6% (= 313 / 12107)

## Papers

### Diffusion Models

* Video Generation
  * Fast Video Generation with Sliding Tile Attention \[[Paper](https://proceedings.mlr.press/v267/zhang25m.html)] \[[OpenReview](https://openreview.net/forum?id=U74MOXPEJd)] \[[arXiv](https://arxiv.org/abs/2502.04507)] \[[Code](https://github.com/hao-ai-lab/FastVideo)]
    * UCSD & UMich & THU & UC Berkeley & MBZUAI
    * Introduce **STA**, a tile-wise local spatiotemporal attention mechanism for video DiTs that removes redundant full attention.
    * Use a hardware-aware sliding-window design and kernel optimizations to speed up video diffusion attention and end-to-end HunyuanVideo generation.


# ATC 2025

## Meta Info

Homepage: <https://www.usenix.org/conference/atc25>

Paper list: <https://www.usenix.org/conference/atc25/technical-sessions>

### Acceptance Rate

15.8% (= 100 / 634)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * GreyHound: Hunting Fail-Slows in Hybrid-Parallel Training at Scale \[[Paper](https://www.usenix.org/conference/atc25/presentation/wu-tianyuan)] \[[Video](https://www.youtube.com/watch?v=R_YsYdY8MPc)] \[[Code](https://github.com/wutianyuan1/Greyhound)]
    * HKUST & Alibaba
    * Takeaways of the characterization study.
      * Fail-slows are usually transient, primarily caused by degradation in computation (slow GPUs or CPU contention) and communication (network congestion).
      * Computation fail-slows tend to be short-lived and less frequent; communication fail-slows due to network congestion are more common and tend to last longer.
      * As training scales up, the likelihood of simultaneously encountering multiple performance issues increases.
    * **GreyHound-Detect**
      * Use the Bayesian online change-point detection (BOCD) algorithm and a verification check to differentiate between real fail-slow issues and normal performance jitters.
      * Change-point verification: compare the average iteration time before and after each identified change-point, treating it as a jitter if the difference is less than 10%.
    * **GreyHound-Mitigate**
      * Ski-rental-like multi-level straggler mitigation → Begin with a low-cost strategy and progressively switch to more effective and more costly strategies if fail-slow persists and the current approach proves ineffective.
      * Adjust the number of micro-batches allocated to DP groups according to their computation performance.
      * Adjust the parallelism topology to reduce congestion and minimize PP stages affected by stragglers.
  * CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training \[[Paper](https://www.usenix.org/conference/atc25/presentation/chen-tiancheng)] \[[Video](https://www.youtube.com/watch?v=V9dXZJukgNY)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_kubicek_ales.pdf)] \[[Code](https://github.com/spcl/crosspipe)]
    * ETH
    * Present a latency and bandwidth-aware performance model designed for the cross-DC environment.
    * PP is better than DP in cross-DC training.
    * Decouple block scheduling from communication arrangement.
  * Obscura: Concealing Recomputation Overhead in Training of Large Language Models with Bubble-filling Pipeline Transformation \[[Paper](https://www.usenix.org/conference/atc25/presentation/huang-yuzhou)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides-huang_yuzhou.pdf)] \[[Video](https://www.youtube.com/watch?v=rygp733JlVU)]
    * SYSU
    * Leverage pipeline transformation to better conceal recomputation overhead.
  * Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation \[[Paper](https://www.usenix.org/conference/atc25/presentation/feng)] \[[Video](https://www.youtube.com/watch?v=vR8rwNyBGYo)]
    * Harvard & ByteDance & USC
    * Schedule the encoder computation within the LLM bubbles.
  * FlexPipe: Maximizing Training Efficiency for Transformer-based Models with Variable-Length Inputs \[[Paper](https://www.usenix.org/conference/atc25/presentation/zhao-hairui)] \[[Video](https://www.youtube.com/watch?v=h0Wqp1pOb90)]
    * Jilin University & UC Riverside
    * Dynamically adjust PP by a live flexibility mechanism.
* LLM Inference
  * DeepServe: Serverless Large Language Model Serving at Scale \[[Paper](https://www.usenix.org/conference/atc25/presentation/hu-junhao)] \[[Video](https://www.youtube.com/watch?v=Ol1g-rn_uNQ)]
    * PKU & Huawei Cloud
  * Weaver: Efficient Multi-LLM Serving with Attention Offloading \[[Paper](https://www.usenix.org/conference/atc25/presentation/gao)] \[[Video](https://www.youtube.com/watch?v=3mYpyoapZZA)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_gao_shiwei.pdf)]
    * THU
    * Opportunity: offload attention from hot to cold instances.
    * Challenge 1: The offloaded attention is blocked by many pre-issued kernels
      * Solution: **GPU-driven control flow**
      * Pipeline: The sender (hot model) writes QKV results in *GPU shared memory* & updates task counter → The receiver (cold model) executes a *polling kernel* to select a task → The receiver writes the output in GPU shared memory & updates completion counter → The sender waits for the output.
    * Challenge 2: The offloaded task is blocked by a single long-running kernel.
      * Solution: **Operator splitting**
      * Pipeline: Sort by the operator's running time → Split the biggest operator into two halves and insert back → Reinsert into the queue & repeat until the waiting time < threshold
  * Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference \[[Paper](https://www.usenix.org/conference/atc25/presentation/li-suyi-toppings)]
    * HKUST & CUHK-SZ & TeleAI & Huawei Cloud
    * A system to serve many LoRA adapters derived from a common base model.
    * Pin the base model on GPUs and dynamically load the requested LoRA adapters from host memory as new requests arrive.
    * Use CPUs to compute the lightweight adaptation for prefilling & switch to the GPUs after loading completes to resume the remaining computation.
    * Schedule heterogeneous LoRA requests to maximize the SLO attainment.
  * QFactory: Accelerating Quantized Large Language Model Serving with Qtile Graphs \[[Paper](https://www.usenix.org/conference/atc25/presentation/zhang-qihao)] \[[Video](https://www.youtube.com/watch?v=zrSYfZUNoGQ)] \[[Artifact](https://github.com/zqh-wz/QFactory-AE)]
    * THU
    * A compilation framework to generate high-performance *quantized kernels*.
    * Transform the traditional tensor computation graph into a Qtile-graph (Qgraph)
    * Explore graph-level Qtile computation transformations to generate equivalent QGraphs.
    * Employ operator-level Qtile scheduling to identify optimal memory loading strategies for each Qtile within the QGraph before generating the final code.
  * CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge \[[Paper](https://www.usenix.org/conference/atc25/presentation/tian)] \[[Video](https://www.youtube.com/watch?v=CNQfMAQOpVs)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides-tian.pdf)]
    * Macau
    * Offline device-specific tailoring
      * LLM layers contribute unevenly to effectiveness and efficiency → Fine-grained layer-wise tuning.
    * Online latency-aware inference
      * Request-wise MoE-based router → Dynamically merge LoRA modules for each mixed-task prompt.
      * Learning-based DVFS (Dynamic Voltage and Frequency Scaling) controller → Reduce per-generated token energy consumption while satisfying the real-time latency target at the layer-wise level.
* LLM Fine-Tuning
  * JENGA: Enhancing LLM Long-Context Fine-tuning with Contextual Token Sparsity \[[Paper](https://www.usenix.org/conference/atc25/presentation/wang-tuowei)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_wang_tuowei.pdf)] \[[Video](https://www.youtube.com/watch?v=6JKYFlXD47g)] \[[Artifact](https://github.com/Pairshoe/Jenga-AE)]
    * THU & MSRA
    * Exploit a new token-level sparsity mechanism inherent in long-context scenarios.
  * mTuner: Accelerating Parameter-Efficient Fine-Tuning on Multi-GPU Servers with Elastic Tensor \[[Paper](https://www.usenix.org/conference/atc25/presentation/huang-kezhao)] \[[Video](https://www.youtube.com/watch?v=J2RmtTfgPqQ)] \[[Code](https://github.com/xxcclong/mTuner)]
    * THU
    * **Elastic Tensor**, an abstraction for dynamic tensor management → Enable flexible control over their availability, accumulation, and release in memory.
* Resource Multiplexing
  * Resource Multiplexing in Tuning and Serving Large Language Models \[[Paper](https://www.usenix.org/conference/atc25/presentation/he-yongjun)] \[[Video](https://www.youtube.com/watch?v=DIohHO_HZgI)] \[[Artifact](https://github.com/llm-db/llmstation/tree/atc25-artifact)]
    * ETH
    * **LLMStation**
      * A new iteration-level multitasking scheduling mechanism.
      * An Autograd engine to transform a tuning task into a suspendable pipeline.
      * An inference engine to batch inference and tuning requests.
* KV Cache Management
  * KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider \[[Paper](https://www.usenix.org/conference/atc25/presentation/wang-jiahao)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_zhang_dingyan.pdf)] \[[Video](https://www.youtube.com/watch?v=A6PuiR-BMis)] \[[Trace](https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon)]
    * SJTU IPADS & Alibaba Cloud
    * Key takeaways from the characterization study
      * KV$ reuses are common, but the reuse ratio is smaller than previously reported numbers on synthetic datasets.
      * For each specific request category, the reuse time is predictable based on the historical information.
      * The lifespan of KV$ is ephemeral.
* SpMM
  * GeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference on GPUs \[[Paper](https://www.usenix.org/conference/atc25/presentation/wang-yaoyu)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides-wang_yaoyu.pdf)] \[[Video](https://www.youtube.com/watch?v=5nWI7d2NR-4)] \[[Code](https://github.com/Wangyaoyuu/GeneralSparse)]
    * ICT, CAS
    * Pruned weight + batch size of dense matrix → SpMM program
  * Voltrix: Sparse Matrix-Matrix Multiplication on Tensor Cores with Asynchronous and Balanced Kernel Optimization \[[Paper](https://www.usenix.org/conference/atc25/presentation/xia)] \[[Video](https://www.youtube.com/watch?v=SPReaFs5_DE)]
    * WHU & NVIDIA & Macau

### Mixture-of-Experts (MoE)

* MoE Training
  * PopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity Based Expert-Wise Prefetch \[[Paper](https://www.usenix.org/conference/atc25/presentation/zhang-junyi)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides-zhang_junyi.pdf)] \[[Video](https://www.youtube.com/watch?v=A0mlhNrv1yg)]
    * HUST
    * Prefetch high-demand experts in the next layer during the execution of current non-MoE computations.
    * Prioritize All-to-All communication stream over All-Reduce operation among prefetched experts.

### Diffusion Models

* Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters \[[Paper](https://www.usenix.org/conference/atc25/presentation/li-suyi-katz)] \[[Video](https://www.youtube.com/watch?v=izS_8clIHvA)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_yang_lingyun.pdf)] \[[Code](https://github.com/modelscope/Katz)] \[[Trace](https://modelscope.cn/datasets/mental2008/T2I-Model-Serving-Request-Trace)]
  * HKUST & Alibaba
  * **Our work!**
  * ControlNet-as-a-Service → Enable ControlNet caching, parallelization, and sharing.
  * Bounded Asynchronous Loading (BAL) → Overlap LoRA loading with initial base model execution by a maximum of K steps.
  * Latent parallelism → Accelerate base model execution across multiple GPUs.

### Deep Learning Recommendation Models (DLRMs)

* DLRM Training
  * Primus: Unified Training System for Large-Scale Deep Learning Recommendation Models \[[Paper](https://www.usenix.org/conference/atc25/presentation/shan-jixi)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_shan_jixi.pdf)]
    * ByteDance
    * Unified resource scheduling
      * Unified API-server upon a diverse cluster.
      * Provide both dynamic horizontal and vertical scaling mechanisms.
      * Standardize YARN & Kubernetes scheduling semantics.
    * Unified data orchestration
      * Support batch and stream data mixture with a three-tier data definition (Dataset, Data Stream, Data Source).
      * Provide a graph-based task planner to accelerate training task generation.
    * Unified training paradigm
      * Mixture Training Recommendation Model (MTRM), a new model with memory and adaptive towers to handle catastrophic forgetting and delayed feedback.

### Deep Learning Compilation

* PluS: Highly Efficient and Expandable ML Compiler with Pluggable Graph Schedules \[[Paper](https://www.usenix.org/conference/atc25/presentation/wu-ruofan)] \[[Video](https://www.youtube.com/watch?v=mXBifQoegGg)]
  * RUC & Microsoft & THU

### GPU Sharing

* Efficient Performance-Aware GPU Sharing with Compatibility and Isolation through Kernel Space Interception \[[Paper](https://www.usenix.org/conference/atc25/presentation/zhang-shulai)] \[[Video](https://www.youtube.com/watch?v=e54BVwcdJ4Y)]
  * SJTU & Lenovo
  * **Krypton**
  * Intercept GPU command buffers at the kernel level to provide virtual GPU devices.
  * The hardware units are divided using MIG, while time slices and device memory are allocated using the kernel-space scheduler.
* GPreempt: GPU Preemptive Scheduling Made General and Efficient \[[Paper](https://www.usenix.org/conference/atc25/presentation/fan)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_fan.pdf)] \[[Video](https://www.youtube.com/watch?v=bFGHPHqT35o)] \[[Code](https://github.com/thustorage/GPreempt)]
  * THU
  * Implement a timeslice-based yield mechanism to enable context-switch preemption on GPUs.
  * Employ a hint-based pre-preemption technique to overlap the preemption process with the essential data-preparation phase.
* Colocating ML Inference and Training with Fast GPU Memory Handover \[[Paper](https://www.usenix.org/conference/atc25/presentation/wang-jiali)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_wang_jiali_0.pdf)] \[[Video](https://www.youtube.com/watch?v=0o6xvkKQDOI)] \[[Code](https://github.com/SiriusInfTra/Sirius)]
  * SJTU IPADS
  * Key insight: training task is elastic and reconfigurable; transfer memory between training and inference by reconfiguring training tasks (i.e., changing batch size).

### Cloud Computing

* Serverless Computing
  * Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference \[[Paper](https://www.usenix.org/conference/atc25/presentation/yu)] \[[Video](https://www.youtube.com/watch?v=a2RUtZCuyyA)] \[[Code](https://github.com/FCSLab/torpor)]
    * CUHK-SZ & HKUST & Alibaba & Nokia Bell Labs
    * Maintain models in main memory and dynamically swap them onto GPUs upon request arrivals.
    * Several techniques to minimize latency overhead caused by model swapping: Asynchronous API redirection, GPU runtime sharing, pipelined model execution, and efficient GPU memory management.
  * Burst Computing: Quick, Sudden, Massively Parallel Processing on Serverless Resources \[[Paper](https://www.usenix.org/conference/atc25/presentation/barcelona-pons)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_barcelona-pons_daniel.pdf)] \[[Video](https://www.youtube.com/watch?v=3T1I0jgWh5o)] \[[Code](https://github.com/Burst-Computing)]

    * Universitat Rovira i Virgili & Barcelona Supercomputing Center
    * Key principle: **group awareness**.
    *

    ```
    <figure><img src="../../.gitbook/assets/Burst-platform-vs-FaaS-platform.png" alt=""><figcaption></figcaption></figure>
    ```
* Image provisioning
  * Poby: SmartNIC-accelerated Image Provisioning for Coldstart in Clouds \[[Paper](https://www.usenix.org/conference/atc25/presentation/chang)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/atc25_slides_chang_zihao.pdf)] \[[Video](https://www.youtube.com/watch?v=xjxGNdodyx4)] \[[Artifact](https://github.com/ACS-Innov/ATC2025-Poby)]
    * ICT, CAS
    * Disaggregated architecture
      * Orchestrate resources of the entire cluster to accelerate image provisioning.
    * Pipeline-based data-driven workflow
      * Pipeline the workflow to enhance efficiency.
      * Eliminate the overhead of control messages.
    * Distributed image download
      * Image metadata index (IMI) records all images and node info.
      * Keep all IMIs in memory.

### Data Preprocessing

* HyCache: Hybrid Caching for Accelerating DNN Input Preprocessing Pipelines \[[Paper](https://www.usenix.org/conference/atc25/presentation/jha)] \[[Video](https://www.youtube.com/watch?v=kZ41xCyAlKM)]
  * IISc & USC
  * Enable the caching of subsets of preprocessed data from multiple intermediate steps on both memory and storage.

### POSIX Shell

* The Koala Benchmarks for the Shell: Characterization and Implications \[[Paper](https://www.usenix.org/conference/atc25/presentation/lamprou)] \[[Video](https://www.youtube.com/watch?v=4YsdRy_S1gA)] \[[Homepage](https://kben.sh/)] \[[Benchmark Suite](https://github.com/kbensh/koala)]
  * Brown University
  * **Best Paper Award**
  * 14 sets of real-world shell programs from diverse domains ranging from CI/CD and AI/ML to biology and the humanities.

## Acronyms

* LoRA: Low-Rank Adaptation
* MoE: Mixture-of-Experts
* PP: Pipeline Parallelism
* DP: Data Parallelism
* SpMM: Sparse-dense Matrix Multiplication


# OSDI 2025

## Meta Info

Homepage: <https://www.usenix.org/conference/osdi25>

Paper list: <https://www.usenix.org/conference/osdi25/technical-sessions>

### Acceptance Rate

14.6% (= 48 / 327)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training \[[Paper](https://www.usenix.org/conference/osdi25/presentation/wang-zheng)] \[[Video](https://www.youtube.com/watch?v=FJqRBCrY8Mg)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_wang_zheng.pdf)] \[[Code](https://github.com/Ash-Zheng/WLB-LLM-CP)]
    * UCSD & Meta
    * Imbalance across DP/PP workers → Input packing
      * Variable-length packing → Balance computation and communication latency.
      * Reorder all documents → Selectively delay long documents.
    * Imbalance across CP workers → Input sharding
      * Adaptively choose the CP sharding strategy with lower latency.
  * ZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization \[[Paper](https://www.usenix.org/conference/osdi25/presentation/wang-zhuang)] \[[Video](https://www.youtube.com/watch?v=G1spkA40_CM)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_wang_zhuang.pdf)]
    * Rice
    * Three techniques for balanced parallelism on GPUs.
      * Communication-oriented hash memory management.
      * Multiple hash functions in each GPU thread.
      * Hierarchical consistent hashing across GPUs.
  * Understanding Stragglers in Large Model Training Using What-if Analysis \[[Paper](https://www.usenix.org/conference/osdi25/presentation/lin-jinkun)] \[[Video](https://www.youtube.com/watch?v=bLr_5OuiUVc)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_lin_jinkun.pdf)] \[[Artifact](https://github.com/ByteDance-Seed/StragglerAnalysis)]
    * NYU & ByteDance Seed
    * Trace
      * 3079 LLM pretraining jobs, collected from *homogeneous* clusters dedicated for LLM training.
    * Common causes of stragglers include: PP stage partitioning imbalance, sequence length imbalance, Python’s garbage collection.
  * Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks \[[Paper](https://www.usenix.org/conference/osdi25/presentation/jiang)] \[[Video](https://www.youtube.com/watch?v=Kf6P6RdGi5k)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_jiang_yuxuan.pdf)] \[[Code](https://github.com/OrderLab/TrainCheck)]
    * UMich
    * **TrainCheck** targets objective correctness violations (e.g., Incorrect API usage, buggy library implementation, faulty hardware).
    * Infer and check *training invariants* to prevent silent training errors.
      * Rule-level training invariants.
        * Example: The weights of certain layers should stay consistent across TP ranks.
      * Instrument a given DL training program to collect traces.
      * Define a set of generic relation templates & generate hypotheses based on a relation template and validate the hypotheses in the traces to generate invariants.
    * Results: Caught 18/20 real-world silent issues, identified 6 new bugs in DeepSpeed and Transformers.
* LLM Inference
  * NanoFlow: Towards Optimal Large Language Model Serving Throughput \[[Paper](https://www.usenix.org/conference/osdi25/presentation/zhu-kan)] \[[Video](https://www.youtube.com/watch?v=Ph7ho4ILQf0)] \[[Code](https://github.com/efeslab/Nanoflow)]
    * UW
    * Split inputs into smaller *nano-batches* and duplicate operations to operate on each portion independently → To overlap heterogeneous operations (e.g., compute, memory, network).
    * Propose *auto-search* to automatically construct an intra-device pipeline.
  * BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching \[[Paper](https://www.usenix.org/conference/osdi25/presentation/zhang-dingyan)] \[[Video](https://www.youtube.com/watch?v=pS5ofuHOVes)] \[[Slides](https://www.usenix.org/system/files/osdi25_slides_zhang_dingyan.pdf)] \[[arXiv](https://arxiv.org/abs/2412.17246)] \[[Code](https://github.com/blitz-serving/blitz-scale)]
    * SJTU IPADS & Huawei Cloud
    * Objective: Optimize model loading to improve instance startup / autoscaling.
    * Two designs
      * Load parameters from remote rather than local cache → Network-based multicast scaling.
        * Employ serial forwarding chain → Parameter multicast is bulk data sequential reading (i.e., *bandwidth bound*) & Limited gains for more complicated multicast algorithms.
        * Fast-link-first greedy forwarding order → Prefer scale-up network (e.g., NVLink) over scale-out network (e.g., RDMA).
        * All-gather model shards by scale-up network to aggregate scale-out network bandwidth.
      * Existing model instances cooperate with newly scaled ones.
        * New GPU instances borrow parameters from old ones (i.e., loading).
        * Old GPU instances borrow computing power from new ones (i.e., multiplexing).
  * WaferLLM: Large Language Model Inference at Wafer Scale \[[Paper](https://www.usenix.org/conference/osdi25/presentation/he)] \[[Slides](https://www.usenix.org/system/files/osdi25_slides_he.pdf)] \[[Video](https://www.usenix.org/system/files/osdi25_slides_he.pdf)] \[[Code](https://github.com/MeshInfra/WaferLLM)]
    * Edinburgh & MSRA
    * Wafer-scale LLM parallelism
    * MeshGEMM: a scalable GEMM algorithm for wafer-scale devices to accelerate the prefill phase.
    * MeshGEMV: a scalable GEMV algorithm for wafer-scale devices to accelerate the decode phase.
* LLM Quantization
  * DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization \[[Paper](https://www.usenix.org/conference/osdi25/presentation/park-yeonhong)] \[[Video](https://www.youtube.com/watch?v=FEStC-7ZlJA)]
    * Seoul National University
    * Store the residual matrix—the difference between full-precision and quantized weights—in CPU, and dynamically fetch the residuals for only a small portion of the weights.

### Deep Learning Compilation

* Performance Profiling
  * KPerfIR: Towards a Open and Compiler-centric Ecosystem for GPU Kernel Performance Tooling on Modern AI Workloads \[[Paper](https://www.usenix.org/conference/osdi25/presentation/guan)] \[[Video](https://www.youtube.com/watch?v=XfXWVgS7icE)] \[[Docs](https://triton-lang.org/main/dialects/ProtonOps.html)] \[[Artifact](https://github.com/ChandlerGuan/kperfir_artifact)]
    * UCSD & Meta & GMU & OpenAI
    * Integrate profiling capabilities directly into the compiler workflow.
    * Two takeaways
      * Performance profiling tools require the compiler’s IR to provide fine-grained performance metrics.
      * Compiler optimization passes need programmable performance profiling tools to effectively guide their optimization decisions.
    * Integrated into the Triton infrastructure.
* Code Generation
  * PipeThreader: Software-Defined Pipelining for Efficient DNN Execution \[[Paper](https://www.usenix.org/conference/osdi25/presentation/cheng)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_cheng_yu.pdf)] \[[Video](https://www.youtube.com/watch?v=ltHR_lS1QM8)] \[[Code](https://github.com/tile-ai/tilelang)]
    * PKU & MSRA
    * Three designs
      * sEU: expose heterogeneous specialized execution units of modern AI accelerators.
      * sTask and sTask-graph: expose fine-grained pipeline parallelism at tile level.
      * Scheduling primitives: build efficient pipeline schedules.
    * Integrated into TileLang.
  * Mirage: A Multi-Level Superoptimizer for Tensor Programs \[[Paper](https://www.usenix.org/conference/osdi25/presentation/wu-mengdi)] \[[Video](https://www.youtube.com/watch?v=CKrQsMHUh8M)] \[[arXiv](https://arxiv.org/abs/2405.05751)] \[[Code](https://github.com/mirage-project/mirage)]
    * CMU
    * µGraphs: a uniform representation of tensor programs at the kernel, thread block, and thread levels of the GPU compute hierarchy.
    * Tensor program → µGraph candidates (via µGraph generator) → verified µGraph (via equivalence verifier) → GPU kernel (via µGraph optimizer)
      * µGraph generator: generate all possible µGraphs up to a bounded size using exhaustive search.
      * Equivalence verifier: check whether generated µGraphs are correct by random testing with theoretical guarantee.
      * µGraph optimizer: apply optimizations that don't affect correctness of µGraphs (e.g., tensor layouts, memory planning, operator scheduling)
  * Bayesian Code Diffusion for Efficient Automatic Deep Learning Program Optimization \[[Paper](https://www.usenix.org/conference/osdi25/presentation/jeong)] \[[Video](https://www.youtube.com/watch?v=5ZL9B8JfGhs)] \[[Code](https://github.com/eai-lab/BayesianCodeDiffusion)]
    * UNIST
    * Reformulate the concepts of prior and posterior distributions in the Bayesian framework to the context of deep learning program optimization.
    * Search for optimal program code in a reduced search space through an iterative diffusion of program code.
    * Implemented in [Ansor](https://www.usenix.org/conference/osdi20/presentation/zheng).
* Transcompiler
  * QiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a Neural-Symbolic Approach \[[Paper](https://www.usenix.org/conference/osdi25/presentation/dong)] \[[Video](https://www.youtube.com/watch?v=0MDa0YlFIzM)]
    * USTC & Cambricon & ICT, CAS & ISCAS
    * Key insight: leverage the code generation ability of LLM to make costly search-based symbolic synthesis computationally tractable.
    * Propose a transcompiler called **QiMeng-Xpiler**, to automatically translate tensor programs across DLS via both LLMs and symbolic program synthesis (i.e., neural-symbolic synthesis).

### GPU

* GPU Kernel Profiling
  * Neutrino: Fine-grained GPU Kernel Profiling via Programmable Probing \[[Paper](https://www.usenix.org/conference/osdi25/presentation/huang-songlin)] \[[Video](https://www.youtube.com/watch?v=Gh1joy8fXww)] \[[Code](https://github.com/open-neutrino/neutrino)]
    * HKU
    * eBPF-inspired probe interface.
      * probe = snippet (i.e., assembly) + tracepoint (at the finest instruction level) + map (*thread-level*: every thread saves, for value profiling & *warp-level*: only warp leader thread saves, for time profiling)
    * Virtualized probe execution model.
      * Directly place probes in the original assembly without protection.
      * Declare an independent register group logically at the assembly level.
    * Implementation
      * A hook driver (in C) to provide runtime support for assembly tracking, code caching, etc.
      * A probe engine (in Python) to instrument parallel assemblies.
      * A DSL compiler (in Python) to translate probes in platform-agnostic Python Tracing DSL into platform-specific assemblies (PTX for CUDA and GCNAsm for ROCm/HIP).
* GPU Preemption
  * Preemptive Scheduling for Diverse XPUs using Multi-level Hardware Model \[[Paper](https://www.usenix.org/conference/osdi25/presentation/shen-weihang)] \[[Code](https://github.com/XpuOS/xsched)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_shen_weihang.pdf)]
    * SJTU IPADS
    * XQueue: An XPU task is abstracted as a sequence of commands executed on a command queue.
    * Multi-level hardware model
      * Level-1: Preempt pending commands (block host CPU from launching new commands, no hardware requirements).
      * Level-2: Preempt in-flight commands (e.g., instruct the μ-controllers to stall command dispatching, leverage command programmability).
      * Level-3: Preempt running commands.
* GPU Communication
  * Enabling Efficient GPU Communication over Multiple NICs with FuseLink \[[Paper](https://www.usenix.org/conference/osdi25/presentation/ren)] \[[Video](https://www.youtube.com/watch?v=SRkM8zMDyf8)]
    * HKUST iSING Lab
    * Integrate high-speed intra-server links as critical extensions of the inter-server network.
    * Implemented as an independent networking module to replace the default Infiniband networking in NCCL.

### Resource Management

* Resource Allocation
  * Decouple and Decompose: Scaling Resource Allocation with DeDe \[[Paper](https://www.usenix.org/conference/osdi25/presentation/xu)] \[[Video](https://www.youtube.com/watch?v=qHEQvMfNrTU)] \[[Code](https://github.com/illinois-nsai/dede)]
    * Harvard & UIUC
    * Decouple entangled resource and demand constraints and decompose the overall optimization into alternating per-resource and per-demand subproblems that can be solved efficiently and in parallel.
    * Released as a Python package.
  * Kamino: Efficient VM Allocation at Scale with Latency-Driven Cache-Aware Scheduling \[[Paper](https://www.usenix.org/conference/osdi25/presentation/domingo)] \[[Video](https://www.youtube.com/watch?v=KoN8KZPIelA)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_david_domingo.pdf)]
    * Rutgers & MSR & Microsoft Azure
    * Objective: Manage VM request latencies for [Protean](https://www.usenix.org/conference/osdi20/presentation/hadary).
    * LatCache Scheduling
      * Key idea: Schedule requests where latency is minimized.
      * ExpectedTime = ProcessingTime + QueueingTime + RemainingTime
* Cold Start
  * Fork in the Road: Reflections and Optimizations for Cold Start Latency in Production Serverless Systems \[[Paper](https://www.usenix.org/conference/osdi25/presentation/chai-xiaohu)] \[[Video](https://www.youtube.com/watch?v=_Q6cXs34U8c)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_chai_xiaohu.pdf)] \[[Trace](https://github.com/antgroup/AFaaS)]
    * Ant Group & THU & SJTU
    * Gaps for fork-based cold start: control-path latency (18-20ms) + resource contention latency (unstable) + user code initialization latency (10ms-1s).
    * AFaaS: Ant FaaS
      * Propose FRI (Function Runtime Interface) to shorten the control path.
      * Resource pooling and sharing to alleviate resource contention.
      * Seeding user code to reduce user code load and initialization.

### Vector Search

* Quake: Adaptive Indexing for Vector Search \[[Paper](https://www.usenix.org/conference/osdi25/presentation/mohoney)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_mohoney_jason.pdf)] \[[Video](https://www.youtube.com/watch?v=8YI6DkhwAgo)] \[[Code](https://github.com/marius-team/quake)]
  * UW-Madison

### Databases

* Tigon: A Distributed Database for a CXL Pod \[[Paper](https://www.usenix.org/conference/osdi25/presentation/huang-yibo)] \[[Slides](https://www.usenix.org/sites/default/files/conference/protected-files/osdi25_slides_huang_yibo.pdf)] \[[Video](https://www.youtube.com/watch?v=oJ2aS7l4Sto)] \[[Code](https://github.com/ut-datasys/tigon)]
  * UT-Austin
  * The first distributed transactional database for a CXL pod.

## Acronyms

* DL: Deep Learning
* DP: Data Parallelism
* CP: Context Parallelism
* PP: Pipeline Parallelism
* TP: Tensor Parallelism
* CXL: Compute Express Link


# ISCA 2025

## Meta Info

Homepage: <https://iscaconf.org/isca2025/>

Paper list: <https://www.iscaconf.org/isca2025/program/>

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models \[[Code](https://zenodo.org/records/15104237)]
    * HKUST-GZ
  * MeshSlice: Efficient 2D Tensor Parallelism for Distributed DNN Training \[[Paper](https://iacoma.cs.uiuc.edu/iacoma-papers/isca25_1.pdf)]
    * UIUC
  * Scaling Llama 3 Training with Efficient Parallelism Strategies
    * Industry Track
* LLM Inference
  * H2-LLM: Hardware-Dataflow Co-Exploration for Heterogeneous Hybrid-Bonding-based Low-Batch LLM Inference
    * Best Paper Nominee
  * SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
  * LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
  * AiF: Accelerating On-Device LLM Inference Using In-Flash Processing
  * LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading
    * UIUC
  * Hybe: GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context Window
  * WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling
  * Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
    * Industry Track
* Retrieval-Augmented Generation (RAG)
  * HeterRAG: Heterogeneous Processing-in-Memory Acceleration for Retrieval-augmented Generation
    * HUST
  * Hermes: Algorithm-System Co-design for Efficient Retrieval Augmented Generation At-Scale
  * RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
* Quantization & Compression
  * Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization
  * Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-Aware Cache Compression
* Performance modeling
  * AMALI: An Analytical Model for Accurately Modeling LLM Inference on Modern GPUs

### Deep Learning Recommendation Models (DLRMs)

* TRACI: Network Acceleration of Input-Dynamic Communication for Large-Scale Deep Learning Recommendation Model

### Resource Management

* GPU Management
  * Forest: Access-aware GPU UVM Management
  * NetCrafter: Tailoring Network Traffic for Non-Uniform Bandwidth Multi-GPU Systems
  * UGPU: Dynamically Constructing Unbalanced GPUs for Enhanced Resource Efficiency
* Serverless Computing
  * Single-Address-Space FaaS with Jord
* Microservices
  * HardHarvest: Hardware-Supported Core Harvesting for Microservices

### Performance Analysis & Benchmark

* Debunking the CUDA Myth Towards GPU-based AI Systems: Evaluation of the Performance and Programmability of Intel's Gaudi NPU for AI Model Serving
* Dynamic Load Balancer in Intel Xeon Scalable Processor: Performance Analyses, Enhancements, and Guidelines
  * UIUC
* DCPerf: An Open-Source, Battle-Tested Performance Benchmark Suite for Datacenter Workloads
  * Industry Track

### AI Chip

* Meta's Second Generation AI Chip: Model-Chip Co-Design and Productionization Experiences
  * Industry Track


# SIGMETRICS 2025

## Meta Info

Homepage: <https://www.sigmetrics.org/sigmetrics2025/>

### Paper List

* <https://www.sigmetrics.org/sigmetrics2025/program.html>
* <https://dl.acm.org/doi/proceedings/10.1145/3726854>

## Papers

### Scheduling

* Improving Multiresource Job Scheduling with Markovian Service Rate Policies \[[Paper](https://dl.acm.org/doi/10.1145/3727117)]
  * UNC & NEU
  * MSR: Markovian Service Rate
* Robust Gittins for Stochastic Scheduling \[[Paper](https://dl.acm.org/doi/10.1145/3726854.3727315)]
  * CMU & Pittsburgh & Microsoft
  * How to design a robust scheduling policy to produce nearly optimal schedules even if there are *modest discrepancies between the predicted distributions and the underlying real distributions*?
  * The standard Gittins index policy is *not robust* in this sense.
* A Gittins Policy for Optimizing Tail Latency \[[Paper](https://dl.acm.org/doi/10.1145/3727109)]
  * Cornell
  * A new scheduling policy to achieve strong tail optimality in the light-tailed M/G/1 with unknown job sizes.
  * The optimal policy turns out to be a variant of the Gittins policy.

### Resource Allocation

* Online Allocation with Multi-Class Arrivals: Group Fairness vs Individual Welfare \[[Paper](https://dl.acm.org/doi/10.1145/3727120)]
  * Alberta & Waterloo
* Allocating Public Goods via Dynamic Max-Min Fairness: Long-Run Behavior and Competitive Equilibria \[[Paper](https://dl.acm.org/doi/10.1145/3711695)]
  * Cornell
* Tight Bounds for Dynamic Bin Packing with Predictions \[[Paper](https://dl.acm.org/doi/10.1145/3700437)]
  * Northwestern & NTU
  * MinUsageTime Dynamic Bin Packing (DBP)
* Online Fair Allocation of Reusable Resources \[[Paper](https://dl.acm.org/doi/10.1145/3727121)]
  * UMass Amherst
* Learning-Augmented Competitive Algorithms for Spatiotemporal Online Allocation with Deadline Constraints \[[Paper](https://dl.acm.org/doi/10.1145/3711701)]
  * UMass Amherst & Caltech & Waterloo & MIT
  * Spatiotemporal Online Allocation with Deadline Constraints (SOAD)

### Serverless Computing

* Exploring Function Granularity for Serverless Machine Learning Application with GPU Sharing \[[Paper](https://dl.acm.org/doi/10.1145/3711699)]
  * NCSU & UCSC
  * Adaptive granularity → Improvement in SLO hit rates & reduction in resource coasts
* PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms \[[Paper](https://dl.acm.org/doi/10.1145/3727125)]
  * SJTU
  * Three techniques
    * Divide each inference services into multiple slices & prewarm slices in a sequential and overlapping manner.
    * Estimate periodic patterns in requests and idle containers for scheduling slices.
    * Incorporate a similarity-based container matcher for the reuse of idle containers.
  * Implemented upon OpenFaaS

### Autoscaling

* Exploiting Kubernetes Autoscaling for Economic Denial of Sustainability \[[Paper](https://dl.acm.org/doi/10.1145/3727114)]
  * Boston University & UMD


# HotOS 2025

## Meta Info

Homepage: <https://sigops.org/s/conferences/hotos/2025/index.html>

Paper list: <https://sigops.org/s/conferences/hotos/2025/program.html>

## Papers

### AI Infrastructure

* Good things come in small packages: Should we build AI clusters with Lite-GPUs? \[[arXiv](https://arxiv.org/abs/2501.10187)]
  * MSR
* Storage Class Memory is Dead, All Hail Managed-Retention Memory: Rethinking Memory for the AI Era \[[arXiv](https://arxiv.org/abs/2501.09605)]
  * MSR
  * **MRM**: Managed-Retention Memory

### Resource Management

* Granular Resource Demand Heterogeneity \[[Paper](https://seojinpark.net/downloads/hotos25-hiresperf.pdf)]
  * USC
  * **hiresperf**: a granular resource profiler that investigates resource usage at *10-microsecond intervals* attributed to each function invocation with a low overhead

### Compound AI Systems

* Towards Resource-Efficient Compound AI Systems \[[arXiv](https://arxiv.org/abs/2501.16634)]
  * MIT & Microsoft Azure

### Operating Systems

* Apiary: An OS for the Modern FPGA
  * UW
* The NIC should be part of the OS
  * ETH

### Tiered Storage

* Tolerate It if You Cannot Reduce It: Handling Latency in Tiered Memory
  * EPFL
* My CXL Pool Obviates Your PCIe Switch \[[arXiv](https://arxiv.org/abs/2503.23611)]
  * Columbia & Microsoft
* Rethinking Tiered Storage: Talk to File Systems, Not Device Drivers
  * UIUC

### Remote Procedure Call (RPC)

* Rethinking RPC Communication for Microservices-based Applications \[[Paper](https://danyangzhuo.com/papers/HotOS25-RPC.pdf)]
  * UW

### AI Security

* Guillotine: Hypervisors for Isolating Malicious AIs \[[arXiv](https://arxiv.org/abs/2504.15499)]
  * Harvard & Princeton

### Verification

* Lightweight Hypervisor Verification: Putting the Hardware Burger on a Diet
  * EPFL
* Can Large Language Models Verify System Software? A Case Study Using FSCQ as a Benchmark
  * Duke
* Modular, Full-System Verification
  * BlueRock Security

### Shared Log

* Designing a Datacenter-wide Distributed Shared Log
  * UC Berkeley

### Unclassified

* Towards ML System Extensibility
  * UW
* Serve Programs, Not Prompts
  * Yale


# MLSys 2025

## Meta Info

Homepage: <https://mlsys.org/Conferences/2025>

Paper list: <https://mlsys.org/virtual/2025/papers.html?filter=titles>

### Acceptance Rate

22.5% (= 61 / 271)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training \[[Paper](https://mlsys.org/virtual/2025/poster/3240)]
    * Cornell & Meta & MIT
  * PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training \[[Paper](https://mlsys.org/virtual/2025/poster/3283)] \[[arXiv](https://arxiv.org/abs/2410.07192)]
    * CMU & AWS
  * Scaling Deep Learning Training with MPMD Pipeline Parallelism \[[Paper](https://mlsys.org/virtual/2025/poster/3257)] \[[arXiv](https://arxiv.org/abs/2412.14374)]
    * NVIDIA
  * Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer \[[Paper](https://mlsys.org/virtual/2025/poster/3268)] \[[arXiv](https://arxiv.org/abs/2408.16978)]
    * OSU & Microsoft
  * APOLLO: SGD-like Memory, AdamW-level Performance \[[Paper](https://mlsys.org/virtual/2025/poster/3241)] \[[Homepage](https://zhuhanqing.github.io/APOLLO/)] \[[arXiv](https://arxiv.org/abs/2412.05270)] \[[Code](https://github.com/zhuhanqing/APOLLO)]
    * UT-Austin & Meta AI
    * **Outstanding Paper Honorable Mention**
  * Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training \[[Paper](https://mlsys.org/virtual/2025/poster/3256)]
    * Rutgers
  * Photon: Federated LLM Pre-Training \[[Paper](https://mlsys.org/virtual/2025/poster/3280)] \[[arXiv](https://arxiv.org/abs/2411.02908)]
    * UCambridge
  * Balancing Pipeline Parallelism with Vocabulary Parallelism \[[Paper](https://mlsys.org/virtual/2025/poster/3237)] \[[arXiv](https://arxiv.org/abs/2411.05288)] \[[Code](https://github.com/sail-sg/VocabularyParallelism)]
    * Sea AI Lab
  * Youmu: Efficient Columnar Data Pipeline for LLM Training \[[Paper](https://mlsys.org/virtual/2025/poster/3272)] \[[Slides](https://mlsys.org/media/mlsys-2025/Slides/3272.pdf)]
    * UVA & UofT & CUHK
* LLM Inference
  * XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models \[[Paper](https://mlsys.org/virtual/2025/poster/3235)] \[[arXiv](https://arxiv.org/abs/2411.15100)] \[[Homepage](https://xgrammar.mlc.ai)] \[[Code](https://github.com/mlc-ai/xgrammar)]
    * CMU & NVIDIA & SJTU & UC Berkeley
  * Seesaw: High-throughput LLM Inference via Model Re-sharding \[[Paper](https://mlsys.org/virtual/2025/poster/3253)] \[[arXiv](https://arxiv.org/abs/2503.06433)]
    * UofT
    * **Outstanding Paper Honorable Mention**
  * NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference \[[Paper](https://mlsys.org/virtual/2025/poster/3230)] \[[arXiv](https://arxiv.org/abs/2411.01142)]
    * Harvard & UC Berkeley
  * FlexInfer: Flexible LLM Inference with CPU Computations \[[Paper](https://mlsys.org/virtual/2025/poster/3234)]
    * GaTech
  * SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling \[[Paper](https://mlsys.org/virtual/2025/poster/3231)]
    * THU
  * Marconi: Prefix Caching for the Era of Hybrid LLMs \[[Paper](https://mlsys.org/virtual/2025/poster/3260)] \[[arXiv](https://arxiv.org/abs/2411.19379)]
    * Princeton & AWS
    * **Outstanding Paper Honorable Mention**
  * Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving \[[Paper](https://mlsys.org/virtual/2025/poster/3236)]
  * QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving \[[Paper](https://mlsys.org/virtual/2025/poster/3288)] \[[arXiv](https://arxiv.org/abs/2405.04532)] \[[Homepage](https://hanlab.mit.edu/projects/qserve)] \[[Code](https://github.com/mit-han-lab/omniserve)]
    * MIT
  * ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments \[[Paper](https://mlsys.org/virtual/2025/poster/3284)] \[[arXiv](https://arxiv.org/abs/2502.09334)]
    * UCambridge & PKU & ETH
  * Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking \[[Paper](https://mlsys.org/virtual/2025/poster/3251)] \[[arXiv](https://arxiv.org/abs/2412.01380)]
    * Qualcomm AI Research
  * Context Parallelism for Scalable Million-Token Inference \[[Paper](https://mlsys.org/virtual/2025/poster/3255)] \[[arXiv](https://arxiv.org/abs/2411.01783)]
    * Meta
  * MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs \[[Paper](https://mlsys.org/virtual/2025/poster/3244)] \[[arXiv](https://arxiv.org/abs/2503.11663)]
    * Yale & IIT Roorkie & IBM Research
* Attention Mechanisms
  * FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving \[[Paper](https://mlsys.org/virtual/2025/poster/3259)] \[[arXiv](https://arxiv.org/abs/2501.01005)] \[[Homepage](https://flashinfer.ai)] \[[Code](https://github.com/flashinfer-ai/flashinfer)]
    * UW & NVIDIA
    * **Outstanding Paper Award**
  * LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention \[[Paper](https://mlsys.org/virtual/2025/poster/3270)] \[[arXiv](https://arxiv.org/abs/2502.14866)] \[[Homepage](https://hanlab.mit.edu/projects/lserve)] \[[Code](https://github.com/mit-han-lab/omniserve)]
    * MIT & NVIDIA
  * FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference \[[Paper](https://mlsys.org/virtual/2025/poster/3278)]
    * UCSD & AWS
  * FlexAttention: A Programming Model for Generating Fused Attention Variants \[[Paper](https://mlsys.org/virtual/2025/poster/3286)] \[[arXiv](https://arxiv.org/abs/2412.05496)]
    * Meta
  * LeanAttention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers \[[Paper](https://mlsys.org/virtual/2025/poster/3269)] \[[arXiv](https://arxiv.org/abs/2405.10480)]
    * Microsoft
  * TurboAttention: Efficient Attention Approximation for High Throughputs LLMs \[[Paper](https://mlsys.org/virtual/2025/poster/3250)] \[[arXiv](https://arxiv.org/abs/2412.08585)]
    * Microsoft & GaTech
  * SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention \[[Paper](https://mlsys.org/virtual/2025/poster/3258)] \[[arXiv](https://arxiv.org/abs/2406.15486)]
    * PKU & CUHK & Zhipu AI & THU & Shanghai AI Lab
* RLHF Training
  * ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation \[[Paper](https://mlsys.org/virtual/2025/poster/3228)] \[[arXiv](https://arxiv.org/abs/2406.14088)] \[[Code](https://github.com/openpsi-project/ReaLHF)]
    * THU
* MoE Inference
  * COMET: Fine-grained Computation-communication Overlapping for Mixture-of-Experts \[[Paper](https://mlsys.org/virtual/2025/poster/3246)] \[[arXiv](https://arxiv.org/abs/2502.19811)]
    * ByteDance Seed & SJTU
    * **Outstanding Paper Honorable Mention**
  * MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators \[[Paper](https://mlsys.org/virtual/2025/poster/3266)] \[[Code](https://github.com/Supercomputing-System-AI-Lab/MiLo)]
    * UIUC
* LoRA Fine-tuning
  * HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression \[[Paper](https://mlsys.org/virtual/2025/poster/3254)] \[[Slides](https://mlsys.org/media/mlsys-2025/Slides/3254_Yq1cyy5.pdf)]
    * THU
* LLM Distillation
  * Self-Data Distillation for Recovering Quality in Pruned Large Language Models \[[Paper](https://mlsys.org/virtual/2025/poster/3247)] \[[arXiv](https://arxiv.org/abs/2410.09982)]
    * Cerebras Systems
* LLM Agent Simulation
  * AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Execution \[[Paper](https://mlsys.org/virtual/2025/poster/3243)] \[[arXiv](https://arxiv.org/abs/2411.03519)]
    * Stanford & GaTech
* LLM for Relational Data Analytics
  * Optimizing LLM Queries in Relational Data Analytics Workloads \[[Paper](https://mlsys.org/virtual/2025/poster/3261)] \[[arXiv](https://arxiv.org/abs/2403.05821)]
    * UC Berkeley

### Diffusion Models

* Video Generation
  * ScaleFusion: Scalable Inference of Spatial-Temporal Diffusion Transformers for High-Resolution Long Video Generation \[[Paper](https://mlsys.org/virtual/2025/poster/3252)]
    * UofT & AWS
* Image Generation
  * DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling \[[Paper](https://mlsys.org/virtual/2025/poster/3287)] \[[arXiv](https://arxiv.org/abs/2411.15381)]
    * UMass Amherst & Adobe Research
    * Construct model cascades → Easy queries can be processed by more lightweight diffusion models

### Resource Management

* Scheduling
  * LAVA: Lifetime-Aware VM Allocation with Learned Distributions and Adaptation to Mispredictions \[[Paper](https://mlsys.org/virtual/2025/poster/3281)] \[[arXiv](https://arxiv.org/abs/2412.09840)]
    * Google
    * **Outstanding Paper Honorable Mention**
  * Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Scheduling \[[Paper](https://mlsys.org/virtual/2025/poster/3232)] \[[arXiv](https://arxiv.org/abs/2408.08586)]
    * ECNU & Alibaba & HUST
* Virtual CPU Oversubscription
  * ProtoRAIL: A Risk-cognizant Imitation Agent for Adaptive vCPU Oversubscription In the Cloud \[[Paper](https://mlsys.org/virtual/2025/poster/3277)] \[[Slides](https://mlsys.org/media/mlsys-2025/Slides/3277_0U2rb6k.pdf)]
    * Microsoft
* AIOps
  * AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds \[[Paper](https://mlsys.org/virtual/2025/poster/3285)] \[[arXiv](https://arxiv.org/abs/2501.06706)]
    * Microsoft

### Deep Learning Compilation

* TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives \[[Paper](https://mlsys.org/virtual/2025/poster/3248)]
  * ByteDance Seed

### Super-Resolution

* VoLUT: Efficient Volumetric streaming enhanced by LUT-based super-resolution \[[Paper](https://mlsys.org/virtual/2025/poster/3275)] \[[arXiv](https://arxiv.org/abs/2502.12151)]
  * UW-Madison & USC & MSRA

### PDF Parsing

* AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine \[[Paper](https://mlsys.org/virtual/2025/poster/3229)] \[[Code](https://github.com/7shoe/AdaParse)] \[[Slides](https://mlsys.org/media/mlsys-2025/Slides/3229.pdf)]
  * UChicago & Argonne National Laboratory

## Acronyms

* RLHF: Reinforcement Learning from Human Feedback
* MoE: Mixture-of-Experts
* LoRA: Low-Rank Adaptation
* LUT: Lookup Table


# NSDI 2025

## Meta Info

Homepage: <https://www.usenix.org/conference/nsdi25>

Paper list: <https://www.usenix.org/conference/nsdi25/technical-sessions>

### Acceptance Rate

* Total: 12.5% (= 83 / 666)
* Fall: 13.7% (= 55 / 401)
* Spring: 10.6% (= 28 / 265)

### Papers

### Large Language Models (LLMs)

* LLM Training
  * Minder: Faulty Machine Detection for Large-scale Distributed Model Training \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/deng)]
    * THU & ByteDance & NEU & Harvard
    * Automatically and efficiently detect faulty distinctive monitoring metric patterns.
  * Holmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/yao)]
    * FDU & Tencent & UChicago
  * Evolution of Aegis: Fault Diagnosis for AI Model Training Cloud Service in Production \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/dong)]
    * Alibaba Cloud
  * Accelerating Design Space Exploration for LLM Training Systems with Multi-experiment Parallel Simulation \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/gui)]
    * THU & Zhongguancun Lab & UPenn
  * SimAI: Unifying Architecture Design and Performance Tunning for Large-Scale Large Language Model Training with Scalability and Precision \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/wang-xizheng-simai)]
    * Alibaba Cloud
* Reinforcement Learning with Human Feedback (RLHF)
  * Optimizing RLHF Training for Large Language Models with Stage Fusion \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/zhong)] \[[arXiv](https://arxiv.org/abs/2409.13221)]
    * PKU & StepFun
* Checkpointing
  * BCP: A Unified Checkpointing System for Large Foundation Model Development \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/wan-borui)]
    * HKU & ByteDance

### Deep Learning Recommendation Models (DLRMs)

* GPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/yang)]
  * HKUST & Alibaba

### Model Serving

* SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/khare)]
  * GaTech & UC Berkeley & Adobe

### Collective Communication

* AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and Parallel DNN Training \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/xu-guanbin)]
  * USTC & Microsoft
* OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/warraich)]
  * Purdue & NVIDIA & VMware Research & Feldera
* Efficient Direct-Connect Topologies for Collective Communications \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/zhao-liangyu)]
  * UW & Raytheon BBN Technologies & MIT

### Networking

* Remote Direct Memory Access (RDMA)
  * White-Boxing RDMA with Packet-Granular Software Control \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/zhao-chenxingyu)]
    * UW & UW-Madison
  * Mitigating Scalability Walls of RDMA-based Container Networks \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/liu-wei)]
    * Alibaba Cloud
* Application Networks
  * High-level Programming for Application Networks \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/zhu)]
    * UW & Duke
* Container Overlay Network
  * ONCache: A Cache-Based Low-Overhead Container Overlay Network \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/lin-shengkai)]
    * SJTU & Broadcom
* Placement
  * Preventing Network Bottlenecks: Accelerating Datacenter Services with Hotspot-Aware Placement for Compute and Storage \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/bazzaz)]
    * Google & USC & Harvard & UCLA & Columbia
* Network Mitigation
  * Enhancing Network Failure Mitigation with Performance-Aware Ranking \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/namyar)]
    * USC & Microsoft

### Resource Management

* Granular Management
  * Quicksand: Harnessing Stranded Datacenter Resources with Granular Computing \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/ruan)]
    * MIT & Brown & USC & VMware Research
      * Provide developers with familiar, high-level abstractions (e.g., data structures, batch computing); decompose them into resource proclets, granular units that each primarily consume resources of one type; split, merge, and migrate resource proclets in milliseconds.
  * GRANNY: Granular Management of Compute-Intensive Applications in the Cloud \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/segarra)]
    * ICL
* Resource Scheduling
  * GREEN: Carbon-efficient Resource Scheduling for Machine Learning Clusters \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/xu-kaiqiang)]
    * HKUST
* Serverless Computing
  * Making Serverless Pay-For-Use a Reality with Leopard \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/cao)]
    * UW-Madison
* Userspace Scheduling
  * The Benefits and Limitations of User Interrupts for Preemptive Userspace Scheduling \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/guo)]
    * UCSD

### Fault Tolerance

* One-Size-Fits-None: Understanding and Enhancing Slow Fault Tolerance in Modern Distributed Systems \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/lu)]
  * UMich & SJTU

### Memory Disaggregation

* Beehive: A Scalable Disaggregated Memory Runtime Exploiting Asynchrony of Multithreaded Programs \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/li-quanxi)]
  * UCAS & PKU & Huawei Cloud & SJTU
* Eden: Developer-Friendly Application-Integrated Far Memory \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/yelam)]
  * UCSD & Technion & VMware Research

### Real-Time Video Streaming

* Mowgli: A Passive Approach to Learning Real-Time Rate Control for Video Conferencing \[[Paper](https://www.usenix.org/conference/nsdi25/presentation/agarwal)]
  * Princeton


# ASPLOS 2025

## Meta Info

Homepage: <https://www.asplos-conference.org/asplos2025/>

### Proceedings

* Volume 1: <https://dl.acm.org/doi/proceedings/10.1145/3669940>
* Volume 2: <https://dl.acm.org/doi/proceedings/10.1145/3676641>

### Acceptance Rate

* Overall: 17.5% (= 160 / 912)
* Fall: 14.1% (= 46 / 326)
  * Major Revision: 20 (invited) -> pending
* Summer: 12.7% (= 65 / 510)
  * Major Revision: 42 (invited) -> 40 (accepted)
* Spring: 2.6% (= 2 / 76)
  * Major Revision: 7 (invited) -> 7 (accepted)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
    * PKU & ByteDance
  * Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
    * PKU & Alibaba
  * Vela: A Virtualized LLM Training System with GPU Direct RoCE
    * IBM Research
  * Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning
    * NUS & GaTech & Alibaba & GMU & SYSU
* LLM Inference
  * Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow \[[arXiv](https://arxiv.org/abs/2406.01566)] \[[Code](https://github.com/Thesys-lab/Helix-ASPLOS25)]
    * CMU
  * Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management
    * Korea University
  * COMET: Towards Practical W4A4KV4 LLMs Serving
    * ICT, CAS
  * Past-Future Scheduler for LLM Serving under SLA Guarantees
    * BUAA & SenseTime
  * POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
    * UW & MSR India
  * Medusa: Accelerating Serverless LLM Inference with Materialization
    * THU
  * vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention \[[arXiv](https://arxiv.org/abs/2405.04437)]
    * MSR India & IIS
  * TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
    * UIUC & Microsoft Azure Research
  * PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System
    * ICT, CAS & ETH & UofT & NVIDIA
  * PIM is All You Need: A CXL-Enabled GPU-Free System for LLM Inference
    * UMich & ETH & Google
  * Fast On-device LLM Inference with NPUs
    * PKU & BUPT
* LLM-based Applications
  * Towards End-to-End Optimization of LLM-based Applications with Ayo
    * CUHK
* MoE Training
  * FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
    * HKUST-GZ & HKUST & HIT-SZ
  * MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
    * HKUST-GZ
* MoE Inference
  * MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
    * UC Berkeley
  * Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
    * SYSU & HKUST & Huawei & Peng Cheng Laboratory
* Retrieval-Augmented Generation (RAG)
  * Accelerating Retrieval-Augmented Generation
    * Cornell & Kansas & UMass Amherst & Samsung Electronics
* Security
  * PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption
    * SJTU IPADS
* Coarse-Grained Reconfigurable Array (CGRA)
  * PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs
    * NYU

### Deep Learning Recommendation Models (DLRMs)

* DLRM Training
  * Frugal: Efficient and Economic Embedding Model Training with Commodity GPUs
    * THU
* DLRM Inference
  * Load and MLP-Aware Thread Orchestration for Recommendation Systems Inference on CPUs
    * Penn State & AMD

### Resource Management

* Shared ML Clusters
  * Design and Operation of Shared Machine Learning Clusters on Campus
    * HKUST
* Resource Oversubscription
  * Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms
    * Microsoft
* Serverless Computing
  * Litmus: Fair Pricing for Serverless Computing
    * Binghamton & Intel Lab
  * Concurrency-Informed Orchestration for Serverless Functions
    * UVA & Alibaba & Amazon
  * Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity
    * ICT, CAS
* Graceful Degradation
  * Cooperative Graceful Degradation in Containerized Clouds \[[arXiv](https://arxiv.org/abs/2312.12809)]
    * UC Irvine
* Microservice
  * Embracing Imbalance: Dynamic Load Shifting among Microservice Containers in Shared Clusters
    * University of Macau

### Deep Learning Compilation

* Mosaic: Exploiting Instruction-Level Parallelism on Deep Learning Accelerators with iTex Tessellation
  * ICT, CAS & Tencent
* Einsum Trees: An Abstraction for Optimizing the Execution of Tensor Expressions
  * Friedrich Schiller University Jena
* Relax: Composable Abstractions for End-to-End Dynamic Machine Learning
  * CMU & NVIDIA
* Pruner: A Draft-then-Verify Exploration Mechanism to Accelerate Tensor Program Tuning
  * USTC & NIO
* Optimizing Deep Learning Inference Efficiency through Block Dependency Analysis
  * ICT, CAS
* Composing Distributed Computations Through Task and Kernel Fusion \[[arXiv](https://arxiv.org/abs/2406.18109)]
  * Stanford & NVIDIA

### Parallelism

* PartIR: Composing SPMD Partitioning Strategies for Machine Learning \[[arXiv](https://arxiv.org/abs/2401.11202)]
  * Google DeepMind
* GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism \[[arXiv](https://arxiv.org/abs/2406.17145)]
  * NVIDIA & CMU & UC Berkeley & MIT

### GPU Sharing

* Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads \[[arXiv](https://arxiv.org/abs/2410.07381)]
  * Stanford & UofT

### Performance Prediction

* Forecasting GPU Performance for Deep Learning Training and Inference
  * GaTech & Meta

### Checkpointing

* PCcheck: Persistent Concurrent Checkpointing for ML
  * ETH

### Memory Disaggregation

* pulse: Accelerating Distributed Pointer-Traversals on Disaggregated Memory
  * Yale
* EDM: An Ultra-Low Latency Ethernet Fabric for Memory Disaggregation
  * Purdue

### Acceleration

* DynaX: Sparse Attention Acceleration with Dynamic X:M Fine-Grained Structured Pruning
  * CQU
* GUST: Graph Edge-Coloring Utilization for Accelerating Sparse Matrix Vector Multiplication
  * UMD
* RASSM: Residue-based Acceleration of Single Sparse Matrix Computation via Adaptive Tiling
  * GaTech
* Squeezing Operator Performance Potential for the Ascend Architecture
  * NJU & Huawei

### Performance Tuning

* DarwinGame: Playing Tournaments for Tuning Applications in Noisy Cloud Environments \[[Paper](https://dl.acm.org/doi/10.1145/3669940.3707259)]
  * Utah & MIT & NEU

### DPU Offloading

* OS2G: A High-Performance DPU Offloading Architecture for GPU-based Deep Learning with Object Storage
  * Alibaba & ZJU

### Tracing

* Automatic Tracing in Task-Based Runtime Systems
  * Stanford & NVIDIA


# EuroSys 2025

## Meta Info

Homepage: <https://2025.eurosys.org>

Paper list: <https://2025.eurosys.org/accepted-papers.html>

Proceedings: <https://dl.acm.org/doi/proceedings/10.1145/3689031>

### Acceptance Rate

* Overall: 12.2% (= 85 / 696)
  * Total: 85 (= 44 + 41)
  * 11 revised papers from EuroSys'25 Fall
* Fall: 8.2% (= 30 / 367)
  * 14 revised papers from EuroSys'25 Spring
  * Total: 44 (= 30 + 14)
* Spring: 9.7% (= 32 / 329)
  * 9 revised papers from EuroSys'24 Fall
  * Total: 41 (= 32 + 9)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
    * UofT
  * MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators
    * THU & Zhipu AI
* LLM Inference
  * Fast State Restoration in LLM Serving with HCache
    * THU
  * Stateful Large Language Model Serving with Pensieve
    * NYU
  * CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion \[[Paper](https://doi.org/10.1145/3689031.3696098)] \[[arXiv](https://arxiv.org/abs/2405.16444)] \[[Code](https://github.com/YaoJiayi/CacheBlend)]
    * CUHK-Shenzhen & UChicago & Stanford
    * **Best Paper Award (Spring)**
    * Reuses multiple precomputed KV caches for RAG inputs and selectively recomputes a small fraction of tokens to recover generation quality.
  * T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
    * USTC & MSRA
  * DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs
    * ETH & MIT
  * SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs
    * HKUST-GZ
    * **Best Paper Award (Fall)**
* LLM Fine-Tuning
  * HybridFlow: A Flexible and Efficient RLHF Framework
    * HKU & ByteDance
* Mixture-of-Experts (MoEs)
  * Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores
    * SJTU

### Distributed Training

* JABAS: Joint Adaptive Batching and Automatic Scaling for DNN Training on Heterogeneous GPUs
  * UNIST & Samsung
* FlowCheck: Decoupling Checkpointing and Training of Large-Scale Models
  * SJTU & Alibaba Cloud
* Comprehensive Deadlock Prevention for GPU Collective Communication
  * PKU & OneFlow

### Model Serving

* A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro
  * UIUC & IBM Research
* SkyServe: Serving AI Models across Regions and Clouds with Spot Instances \[[Paper](https://dl.acm.org/doi/10.1145/3689031.3717459)] \[[Code](https://github.com/skypilot-org/skypilot)] \[[arXiv](https://arxiv.org/abs/2411.01438)]
  * UC Berkeley
  * Manage a mixture of spot and on-demand replicas across regions and clouds.
  * Improve availability, reduce correlated preemptions, overprovision cheap spot replicas.
  * Baselines: AWS Auto-scaling Group (ASG), MArk \[ATC'19], AWS spot node pool (AWSSpot), [SpotServe](/reading-notes/conference/asplos-2024/spotserve)

### Deep Learning Compilation

* SpaceFusion: Advanced Deep Learning Operator Fusion via Space-Mapping Graph
  * SJTU

### Resource Management

* Scheduling
  * Towards VM Rescheduling Optimization Through Deep Reinforcement Learning
    * UC Merced & UC Berkeley & ByteDance
  * Eva: Cost-Efficient Cloud-Based Cluster Scheduling
    * UW-Madison
* Serverless Computing
  * Serverless Cold Starts and Where to Find Them
    * Huawei
  * SeBS-Flow: Benchmarking Serverless Cloud Function Workflows
    * Karlsruhe Institute of Technology & ETH
  * AlloyStack: A Library Operating System for Serverless Workflow Applications
    * TJU & THU
* GPU Sharing
  * Improving GPU Sharing Performance through Adaptive Bubbleless Spatial-Temporal Sharing
    * SJTU & Microsoft & Alibaba
  * Multiplexing Dynamic Deep Learning Workloads with SLO-awareness in GPU Clusters
    * University of Macau & SIAT, CAS

## Acronyms

* RLHF: Reinforcement Learning from Human Feedback
* ML: Machine Learning


# HPCA 2025

## Meta Info

Homepage: <https://hpca-conf.org/2025/>

Paper list: <https://hpca-conf.org/2025/main-program/>

### Acceptance Rate

21% (= 112 / 534)

## Papers

### Large Language Models (LLMs)

* LLM Compression
  * eDKM: An Efficient and Accurate Train-Time Weight Clustering for Large Language Models \[[arXiv](https://arxiv.org/abs/2309.00964)]
    * Apple
    * Compress LLMs to fit into storage-limited devices.
    * Propose a memory-efficient DKM (Differentiable KMeans Clustering) implementation.
    * Fine-tune and compress a pretrained LLaMA 7B model from 12.6 GB to 2.5 GB.
* LLM Quantization
  * BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration \[[arXiv](https://arxiv.org/abs/2411.11745)] \[[Code](https://github.com/yc2367/BitMoD-HPCA-25)]
    * Cornell & MSR & ICL
    * Algorithm: Fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights.
    * Hardware: Employ a bit-serial processing element to support multiple numerical precisions and data types.
  * MANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type \[[arXiv](https://arxiv.org/abs/2502.18755)]
    * SJTU
    * Assign the appropriate data type for each group adaptively.
    * Propose an efficient real-time quantization mechanism.
    * Implement a specific processing element to efficiently support MANT and incorporate a real-time quantization unit.
  * Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format \[[arXiv](https://arxiv.org/abs/2411.15982)]
    * NJU & MICAS KU Leuven
    * Investigate the sensitivity of activation precision across various LLM modules and its impact on overall model accuracy.
    * Anda data type: an adaptive data format with group-shared exponent bits and dynamic mantissa bit allocation.
  * VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
    * SJTU
* Energy-Efficient LLM Inference
  * DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency \[[arXiv](https://arxiv.org/abs/2408.00741)]
    * UIUC & Microsoft Azure
    * Given the current load and available resources, select the energy-optimized configuration (e.g., different model parallelisms, GPU frequencies).
  * throttLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving
    * National Technical University of Athens
* Long-Context LLM Inference
  * InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference \[[arXiv](https://arxiv.org/abs/2409.04992)]
    * PKU
    * Offload the attention in decoding phase KV cache to Computational Storage Drives (CSDs).
* Hardware-Assisted LLM Inference
  * LAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
    * ICT, CAS
  * PAISE: PIM-Accelerated Inference Scheduling Engine for Transformer-based LLM
    * Samsung SDS
  * FACIL: Flexible DRAM Address Mapping for SoC-PIM Cooperative On-device LLM Inference
    * Seoul National University
  * Lincoln: Real-Time 50\~100B LLM Inference on Consumer Devices with LPDDR-Interfaced, Compute-Enabled Flash Memory
    * THU
  * Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM \[[arXiv](https://arxiv.org/abs/2502.16963)]
    * ICT, CAS
    * **Hermes**

### Diffusion Models

* EXION: Exploiting Inter- and Intra-Iteration Output Sparsity for Diffusion Models \[[arXiv](https://arxiv.org/abs/2501.05680)]
  * KAIST
  * Inter-iteration sparsity → a FFN-Reuse algorithm to identify and skip redundant computations in FFN layers across different iterations
  * Intra-iteration sparsity → A modified eager prediction method to accurately predict the attention score, skipping unnecessary computations within an iteration
  * A dedicated hardware architecture to support the sparsity-inducing algorithms.
* Ditto: Accelerating Diffusion Model via Temporal Value Similarity \[[arXiv](https://arxiv.org/abs/2501.11211)]
  * Yonsei University
  * Ditto: a difference processing algorithm
    * Leverage temporal similarity with quantization.
    * Perform full bit-width operations for the initial time step and process subsequent steps with temporal differences.
  * Design the Ditto hardware → a specialized hardware accelerator

### Deep Learning Recommendation Models (DLRMs)

* Machine Learning-Guided Memory Optimization for DLRM Inference on Tiered Memory
  * UC Merced & Meta

### Dynamic Neural Networks

* Adyna: Accelerating Dynamic Neural Networks with Adaptive Scheduling
  * THU

### ML Cluster Reliability

* Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimization \[[arXiv](https://arxiv.org/abs/2406.04594)]
  * Alibaba
* Revisiting Reliability in Large-Scale Machine Learning Research Clusters \[[arXiv](https://arxiv.org/abs/2410.21680)]
  * Meta

### ML for Systems

* The Importance of Generalizability in Machine Learning for Systems
  * MIT & Google

### ML Benchmark

* MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from μWatts to MWatts for Sustainable AI \[[arXiv](https://arxiv.org/abs/2410.12032)]

### Multi-GPU Systems

* OASIS: Object-Aware Page Management for Multi-GPU Systems \[[Paper](https://users.elis.ugent.be/~leeckhou/papers/HPCA2025-OASIS.pdf)]
  * Pittsburgh & NVIDIA & Ghent

### Collective Communication

* TidalMesh: Topology-Driven AllReduce Collective Communication for Mesh Topology
  * KAIST
* PIMnet: A Domain-Specific Network for Efficient Collective Communication in Scalable PIM
  * KAIST & Northeastern University & Boston University

### Interconnect

* Push Multicast: A Speculative and Coherent Interconnect for Mitigating Manycore CPU Communication Bottleneck \[[Code](https://zenodo.org/records/14355343)]
  * HKUST-GZ & Intel & UCSD & HKUST
* EIGEN: Enabling Efficient 3DIC Interconnect with Heterogeneous Dual-Layer Network-on-Active-Interposer
  * FDU

### Compute Express Link (CXL)

* SkyByte: Architecting An Efficient Memory-Semantic CXL-based SSD with OS and Hardware Co-design \[[arXiv](https://arxiv.org/abs/2501.10682)]
  * UIUC
  * Opportunistic context switches upon the detection of long access delays.
  * Data coalescing upon log cleaning to reduce the I/O traffic to flash chips.
  * Adaptive page migration to promote hot pages in CXL-SSD to the host.
  * Implemented with a CXL-SSD simulator.

### Near-Memory Processing

* AsyncDIMM: Achieving Asynchronous Execution in DIMM-Based Near-Memory Processing
  * SJTU
* UniNDP: A Unified Compilation and Simulation Tool for Near DRAM Processing Architectures \[[Code](https://github.com/UniNDP-hpca25-ae/UniNDP)]
  * THU & HKUST & PKU

### Bandwidth Partitioning

* Criticality-Aware Instruction-Centric Bandwidth Partitioning for Data Center Applications
  * PKU

### Deep Learning Accelerator

* LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator \[[arXiv](https://arxiv.org/abs/2501.10658)]
  * MSRA & NTU
  * Transform various DNN models into LUTs (Look-Up Tables) via multistage training to achieve extreme low-bit quantization.
* FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables
  * POSTECH & NAVER

### Image Signal Processor

* IRIS: Unleashing ISP-Software Cooperation to Optimize the Machine Vision Pipeline
  * Universitat Politècnica de Catalunya
  * Enable the Image Signal Processor (ISP) to create compressed, mixed-resolution images in real time.
  * Used for image classification & monocular Simultaneous Localization and Mapping (SLAM).

### Microservice

* Grad: Intelligent Microservice Scaling by Harnessing Resource Fungibility
  * University of Macau
  * Profile individual microservice latency in relation to environmental conditions.
  * Dynamically select the optimal set of microservices for scaling.
  * An end-to-end latency predictor serves as a simulator to obtain real-time feedback.

## Acronyms

* ML: Machine Learning
* DKM: Differentiable KMeans Clustering
* CXL: Compute Express Link
* CSD: Computational Storage Drive
* LUT: Look-Up Table
* ISP: Image Signal Processor
* SLAM: Simultaneous Localization and Mapping


# PPoPP 2025

## Meta Info

Homepage: <https://ppopp25.sigplan.org>

Paper list: <https://ppopp25.sigplan.org/track/PPoPP-2025-Main-Conference-1#event-overview>

### Acceptance Rate

20.1% (= 38 / 189)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training
    * Oregon & Pacific Northwest National Laboratory & William and Mary
  * Mario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism
    * ICT, CAS
  * WeiPipe: Weight Pipeline Parallelism for Communication-Effective Long-Context Large Model Training
    * THU & NUS & CETHIK & Lynxi Technology
* LLM Inference
  * MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
    * ISTA & Universidade da Coruña & ETH & IST Austria

### Mixture-of-Experts (MoEs)

* MoE Training
  * Harnessing Inter-GPU Shared Memory for Seamless MoE Communication-Computation Fusion
    * WHU & NVIDIA & UMacau

### Graph Neural Networks (GNNs)

* GNN Training
  * Adaptive Parallel Training for Graph Neural Networks \[[Code](https://anonymous.4open.science/r/APT-1CAB)]
    * CUHK
* GNN Inference
  * Helios: Efficient Distributed Dynamic Graph Sampling for Online GNN Inference
    * ZJU & Alibaba

### GPU Sharing

* SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs
  * HKUST

### Sparse Matrix-Matrix Multiplication (SpMM)

* Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Cores
  * Computer Network Information Center, CAS & RUC & Hangzhou Dianzi University
* FlashSparse: Minimizing Computation Redundancy for Fast Sparse Matrix Multiplications on Tensor Cores
  * BUPT


# NeurIPS 2024

## Meta Info

Homepage: <https://neurips.cc/Conferences/2024>

Paper list: <https://neurips.cc/virtual/2024/papers.html?filter=titles>

### Acceptance Rate

* Total: 15671
* Accept: 25.8% (4037)
  * Poster: 23.3% (3650)
  * Spotlight: 2.1% (326)
  * Oral: 0.4% (61)

## Papers

### Large Language Models (LLMs)

* LLM Inference
  * SGLang: Efficient Execution of Structured Language Model Programs \[[Paper](https://openreview.net/forum?id=VqkAKQibpq)] \[[Code](https://github.com/sgl-project/sglang)] \[[arXiv](https://arxiv.org/abs/2312.07104)]
    * Stanford & UC Berkeley
    * Co-design both the front-end language (programming interface) and the back-end runtime
    * SGLang Primitives
      * Enable the manipulation of prompts and generations
        * `gen`: call LLM generation
        * `select`: let the LLM choose the option with the highest probability from a list
        * `extend` or `+=`: extend the current prompt
      * Control of parallelism
        * `fork`: fork the current prompt state
        * `join`: rejoin the forked prompt states
    * Compilation optimizations
      * Code movement for improving prefix sharing
        * Doesn't strictly preserve the original computation —— aggressive
        * Prompt GPT-4 to re-order graph nodes
    * Runtime
      * RadixAttention
        * Utilize a radix tree (w/ efficient prefix search, reuse, insertion, eviction)
        * LRU eviction policy
      * Cache-aware scheduling → Increase the cache hit rate
        * Key idea: Sort the requests by matched prefix length
  * Efficient LLM Scheduling by Learning to Rank \[[Paper](https://openreview.net/forum?id=wlLjYl0Gi6)] \[[Code](https://github.com/hao-ai-lab/vllm-ltr)]
    * UCSD & THU & Snowflake & UC-Berkeley
    * Insight: it is possible to *predict the relative ranks of output lengths in a batch of requests*.
    * Develop a scheduler for LLM inference that can approximate the *shortest-job-first* (SJF) schedule better than existing approaches
* Compound AI Systems
  * Are More LM Calls All You Need? Towards the Scaling Properties of Compound AI Systems \[[Paper](https://openreview.net/forum?id=m5106RRLgx)] \[[Code](https://github.com/lchen001/CompoundAIScalingLaws)]
    * Stanford & UC Berkeley & Princeton
    * Systematically study *how the number of LM calls affects the performance of two natural inference strategy designs*.
      * **Vote**: Aggregate LM responses via majority voting
      * **Filter-Vote**: Majority voting after filtering results with an LM
    * Insight
      * More LM calls lead to higher performance on “easy” queries, but lower performance on “hard” queries, and nonmonotone behavior can emerge when a task contains both types of queries.
    * An analytical scaling model to predict the performance of Vote and Filter-Vote systems and find the optimal number of LM calls to make.

### Diffusion Models

* Adapter Selection
  * Stylus: Automatic Adapter Selection for Diffusion Models \[[Paper](https://openreview.net/forum?id=3Odq2tGSpp)] \[[Homepage](https://stylus-diffusion.github.io)] \[[Code](https://github.com/stylus-diffusion/stylus)]
    * UC Berkeley & CMU & Google DeepMind
    * Problem: how to match the prompt to a set of relevant adapters
    * Stylus
      * Select and automatically compose task-specific adapters based on a prompt's keywords
      * Three-stage approach
        1. Refiner: Leverage visual-language foundational models (VLM) to generate semantic descriptions of adapters then translate them into embeddings
        2. Retriever: Fetch the most relevant adapters over the entirety of the user’s prompt using cosine similarity
        3. Composer: Segment the prompt into tasks from a prompt’s keywords and assign retrieved adapters to tasks
    * StylusDocs
      * An adapter dataset consists of 75K LoRAs (sourced from Civitai) with pre-computed adapter embeddings
* Inference
  * Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference \[[Paper](https://openreview.net/forum?id=C2xCLze1kS)]
    * HKUST & HKU & Salesforce AI Research & UIUC
    * Develop a general RTK (reverse transition kernel) framework that enables a more balanced subproblem decomposition
    * Propose leveraging two fast sampling algorithms, the Metropolis-Adjusted Langevin Algorithm (MALA) and Underdamped Langevin Dynamics (ULD), for solving these strongly log-concave subproblems
  * Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity \[[Paper](https://openreview.net/forum?id=F9NDzHQtOl)]
    * Stanford
    * Propose to divide the sampling process into $$O(1)$$ blocks with parallelizable Picard iterations within each block
* Talking Face Video Generation
  * VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time \[[Paper](https://openreview.net/forum?id=5zSCSE0k41)] \[[Homepage](https://www.microsoft.com/en-us/research/project/vasa-1/)]
    * MSRA
    * A framework to generate lifelike talking faces with appealing visual affective skills (VAS).
    * A diffusion-based holistic facial dynamics and head movement generation model that works in a face latent space.
    * Support the online generation of 512×512 videos at up to 40 FPS.
* Facial Parts Swapping
  * FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images \[[Paper](https://openreview.net/forum?id=X2UMdvcmMo)] \[[Code (coming...)](https://github.com/Thomas-wyh/FuseAnyPart)]
    * Alibaba
    * Facial parts from different people are assembled into a complete face in latent space within the Mask-based Fusion Module
    * The consolidated feature is dispatched to the Addition-based Injection Module for fusion within the UNet of the diffusion model to create novel characters

### Autoregressive Image Generation

* Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction \[[Paper](https://openreview.net/forum?id=gojL67CfS8)] \[[Code](https://github.com/FoundationVision/VAR)] \[[arXiv](https://arxiv.org/abs/2404.02905)]
  * PKU & ByteDance
  * **Best Paper Award**
  * **VAR**: Visual Autoregressive Modeling
  * Redefine the autoregressive learning on images as coarse-to-fine “**next-scale prediction**” or “**next-resolution prediction**”
  * Multi-scale token maps are autoregressively generated from coarse to fine scales (**lower to higher resolutions**), with parallel token generation within each scale
* Autoregressive Image Generation without Vector Quantization \[[Paper](https://openreview.net/forum?id=VNBIF0gmkb)] \[[Code](https://github.com/LTH14/mar)] \[[arXiv](https://arxiv.org/abs/2406.11838)]
  * MIT & Google DeepMind & THU
  * Propose to model the per-token probability distribution using a diffusion procedure
  * Define a *Diffusion Loss* function to model the per-token probability
  * Evaluated across a wide range of cases, including standard autoregressive models and generalized *masked autoregressive* (**MAR**) variants

### Text-to-Video Generation

* Inference
  * Fast and Memory-Efficient Video Diffusion Using Streamlined Inference \[[Paper](https://openreview.net/forum?id=iNvXYQrkpi)] \[[Code](https://github.com/wuyushuwys/FMEDiffusion)]
    * NEU
    * **Streamlined Inference**: Leverage the temporal and spatial properties of video diffusion models
    * Three core components
      * **Feature Slicer**: Partition input features into sub-features
      * **Operator Grouping**: Process each sub-feature with a group of consecutive operators
      * **Step Rehash**: Accelerate inference through skipping unnecessary steps
* Evaluation
  * VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models \[[Paper](https://openreview.net/forum?id=pYNl76onJL)] \[[Homepage](https://vidprom.github.io)] \[[Code](https://github.com/WangWenhao0716/VidProM)] \[[Dataset](https://huggingface.co/datasets/WenhaoWang/VidProM)]
    * UTS & ZJU
    * 1.67M unique text-to-video Prompts from real users.
    * 6.69M videos generated by four state-of-the-art diffusion models (Pika, VideoCraft2, Text2Video-Zero, ModelScope).
  * Evaluation of Text-to-Video Generation Models: A Dynamics Perspective \[[Paper](https://openreview.net/forum?id=tmX1AUmkl6)] \[[Homepage](https://t2veval.github.io/DEVIL/)] \[[Code](https://github.com/MingXiangL/DEVIL)]
    * UCAS & HIT & Adelaide & Baidu
      * Existing evaluation protocols primarily focus on temporal consistency and content continuity, yet largely *ignore the dynamics of video content*.
      * **DEVIL**: An evaluation protocol that centers on the dynamics dimension to evaluate T2V generation models
  * Boosting Text-to-Video Generative Model with MLLMs Feedback \[[Paper](https://openreview.net/forum?id=3ivnixHy16)]
    * MSRA
      * Utilize Multimodal Large Language Models (MLLMs) to perform fine-grained video preference annotations → **VideoPrefer** (13.5K preference annotations)
      * **VideoRM**: The reward model for text-to-video alignment


# SoCC 2024

## Meta Info

Homepage: <https://acmsocc.org/2024/index.html>

Paper list: <https://acmsocc.org/2024/schedule.html>

### Acceptance Rate

30.1% (= 63 / 209)

## Papers

### Large Language Models (LLMs)

* LLM inference
  * Queue Management for SLO-Oriented Large Language Model Serving \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698523)]
    * UIUC & IBM Research
* LLM training
  * Distributed Training of Large Language Models on AWS Trainium \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698535)]
    * AWS

### Mixture of Experts (MoEs)

* MoE inference
  * MoEsaic: Shared Mixture of Experts \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698521)]
    * IBM Research

### GPU Sharing

* KACE: Kernel-Aware Colocation for Efficient GPU Spatial Sharing \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698555)]
  * Stony Brook University

### Serverless Computing

* On-demand and Parallel Checkpoint/Restore for GPU Applications \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698563)]
  * SJTU IPADS & Shanghai Artificial Intelligence Research Institute
  * **gCROP**: **G**PU **C**heckpoint/**R**estore made **O**n-demand and **P**arallel

### Resource Scheduler

* Scheduler for deep learning training workloads
  * Hops: Fine-grained heterogeneous sensing, efficient and fair Deep Learning cluster scheduling system \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698515)]
    * Anhui University & Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

### Distributed Training

* Generative Adversarial Networks (GANs)
  * ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks \[[Paper](https://dl.acm.org/doi/10.1145/3698038.3698563)]
    * NUS


# HotNets 2024

## Meta Info

Homepage: <https://conferences.sigcomm.org/hotnets/2024/index.html>

Paper list: <https://conferences.sigcomm.org/hotnets/2024/program.html>

## Papers

### Large Language Models (LLMs)

* Networking for LLM training
  * I’ve Got 99 Problems But FLOPS Ain’t One \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-333.pdf)]
    * University Politehnica of Bucharest
    * The future of large-scale AI infrastructure requires
      * (1) novel wide-area transports for inter-DC communication;
      * (2) a multipath transport and novel datacenter topologies for intra-datacenter communication;
      * (3) high-speed scale-up networks and transport.
* LLM for networking
  * Designing Network Algorithms via Large Language Models \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-88.pdf)]
    * MSR
    * **NADA**: Network Algorithm Design Automation via LLMs

### Congestion Control

* MLTCP: A Distributed Technique to Approximate Centralized Flow Scheduling For Machine Learning \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-166.pdf)]
  * MIT
  * Scale the congestion window size (or sending rate) based on the number of bytes sent at each iteration.

### Trading Systems

* Network Design Considerations for Trading Systems \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-262.pdf)]
  * Jane Street

### Caching

* Revisiting Cache Freshness for Emerging Real-Time Applications \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-21.pdf)]
  * UC Berkeley
  * At real-time timescales, making freshness decisions in response to incoming writes is more efficient than TTL-based policies.
* Rethinking Web Caching: An Optimization for the Latency-Constrained Internet \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-124.pdf)]
  * Shahid Beheshti University & Università della Svizzera italiana & Institute for Research in Fundamental Sciences & Sharif University of Technology
  * Web servers *proactively* provide clients with the latest validation tokens for resources during the initial step of page loading → Allow browsers to use unchanged cached content without unnecessary round trips.

### Performance Analysis

* End-to-End Performance Analysis of Learning-enabled Systems \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-138.pdf)]
  * USC & Hebrew University of Jerusalem & Rice University & Microsoft
  * A gray-box approach (leverage partial information) → Use gradient to analyze the performance of DNNs.
* Buffy: A Formal Language-Based Framework for Network Performance Analysis \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-2.pdf)]
  * UWaterloo & Princeton
  * Language abstractions to enable users to model network functionality and analysis tasks in an imperative solver-agnostic program.
  * A framework to transform them into a representation that can be analyzed by the appropriate solver.

### Reliability

* Automatic Configuration Repair \[[Paper](https://conferences.sigcomm.org/hotnets/2024/papers/hotnets24-398.pdf)]
  * XJTU & ByteDance
  * Draw some insights from the field of Automatic Software Repair (ASR).
  * Propose **localize-fix-validate** as a possible approach to realize **Automatic Configuration Repair (ACR)**.

## Acronyms

* TTL: Time-To-Live
* DNN: Deep Neural Network
* DC: Datacenter


# SC 2024

## Meta Info

Homepage: <https://sc24.conference-program.com>

Paper list: <https://dl.acm.org/doi/proceedings/10.5555/3703596>

## Papers

### AI Infrastructure

* Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00089)] \[[HAI Platform Code](https://github.com/HFAiLab/hai-platform)]
  * DeepSeek AI
  * Include Network Co-Design, HFReduce (collective communication library), HaiScale (optimized parallelism methods), 3FS Distributed File System, and HAI Platform (task scheduling, fault tolerance).

### Large Language Models (LLMs)

* LLM inference
  * PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00046)] \[[Code](https://github.com/AutonomicPerfectionist/PipeInfer)]
    * Iowa State University & TU Darmstadt
    * *Continuous Asynchronous Speculation*: run single-token inference simultaneously with several speculative runs.
    * *Early Inference Cancellation*: skip the computation of invalidated runs.
  * LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00022)] \[[Benchmark](https://github.com/fmperf-project/fmperf)] \[[Code](https://github.com/IBM/LLM-performance-prediction)]
    * IBM Research
    * Learn a predictive model to recommend the most cost-effective hardware for a previously unseen LLM.
* LLM fine-tuning
  * Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00081)] \[[Code](https://github.com/HPHEX/LongExposure)]
    * MSRA & THU
* LLM for anomaly detection
  * Large Language Models for Anomaly Detection in Computational Workflows: From Supervised Fine-Tuning to In-Context Learning \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00098)] \[[Code](https://github.com/PoSeiDon-Workflows/LLM_AD)] \[[Benchmark](https://github.com/PoSeiDon-Workflows/FlowBench)]
    * Argonne National Laboratory & USC & Oak Ridge National Laboratory
    * Investigated two approaches: (1) supervised fine-tuning (pre-trained LLMs are fine-tuned on labeled data for sentence classification to identify anomalies); (2) in-context learning (prompts containing task descriptions and examples guide LLMs in few-shot anomaly detection without fine-tuning).

### Mixture-of-Experts (MoEs)

* APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00096)] \[[Code](https://github.com/Atopos-309/APTMoE)]
  * SYSU

### Deep Learning Recommendation Models (DLRMs)

* Accelerating Distributed DLRM Training with Optimized TT Decomposition and Micro-Batching \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00055)] \[[Code](https://doi.org/10.5281/zenodo.13324403)]
  * WHU & NVIDIA & UMacau
  * **EcoRec:** eliminate redundancy in TT (Tensor-Train) operations; micro-batching with sorted indices to reduce memory.
* Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00095)] \[[Code](https://zenodo.org/records/13119689)]
  * Indiana University, Bloomington & Meta & University of Rochester & ICT, CAS
  * In-depth analysis of embedding data features; employ error-bounded lossy compression to reduce the communication data size.
* Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00100)] \[[Code](https://github.com/luckyq/ADSC-24)]
  * UC Merced & SK Hynix
  * **TECO**: Tensor-CXL-Offload
  * Introduce a cache coherence interconnect based on CXL to build a cache coherence domain between CPU memory and accelerator memory; offload tensors to CPU memory to save accelerator memory.
* RecFlex: Enabling Feature Heterogeneity-Aware Optimization for Deep Recommendation Models with Flexible Schedules \[[Paper](https://dl.acm.org/doi/pdf/10.1109/SC41406.2024.00047)] \[[Code](https://github.com/PanZaifeng/RecFlex)]
  * RUC & Microsoft & UCSD
  * Create fused kernels with distinct schedules for *different* feature fields.

### Graph Transformer

* TorchGT: A Holistic System for Large-Scale Graph Transformer Training \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00083)] \[[Code](https://github.com/zxmeng98/torchgt)]
  * NTU & Shanghai AI Lab & ZJU & SenseTime

### **Reinforcement Learning (RL)**

* Stellaris: Staleness-Aware Distributed Reinforcement Learning with Serverless Computing \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00045)] \[[Code](https://github.com/IntelliSys-Lab/Stellaris-SC24)]
  * Stevens Institute of Technology & NEU & Stony Brook University & Missouri University of Science and Technology
  * Introduce a generic asynchronous learning paradigm.

### Job Scheduling

* PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00032)] \[[Code](https://github.com/hal-uw/blox-pal)]
  * UW-Madison
  * Characterize which applications are more likely to suffer from performance variability; balance performance variability with locality to ensure jobs are spread across as few nodes as possible.

### Distributed Training

* ,Optimizing Distributed ML Communication with Fused Computation-Collective Operations \[[Paper](https://dl.acm.org/doi/pdf/10.1109/SC41406.2024.00094)]
  * AMD
  * Developed three prototype fused operators (embedding + All-to-All, GEMV + AllReduce, and GEMM + All-to-All) to address the communication overheads in DLRM, Transformers and MoE model architectures.

### Serverless Computing

* SMIless: Serving DAG-based Inference with Dynamic Invocations under Serverless Computing \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00044)] \[[Code](https://github.com/blinkbear/smiless-ad)]
  * SIAT, CAS & UMacau
  * Integrate adaptive pre-warming windows; built on top of OpenFaaS.

### GPU Sharing

* ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00048)] \[[Code](https://github.com/MunQ-Lee/ParvaGPU_SC24)]
  * Chung-Ang University & Electronics and Telecommunications Research Institute & Virginia Tech
  * Integrate MIG and MPS to enhance GPU utilization.

### Performance Analysis

* GVARP: Detecting Performance Variance on Large-Scale Heterogeneous Systems \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00063)] \[[Code](https://zenodo.org/records/10975567)]
  * BUAA
  * Employ *static analysis* to identify the performance-critical parameters of kernel functions; segment the program execution with external library calls and asynchronous kernel operations; construct a state transfer graph and estimate the workload of each program segment.

### Interconnects

* Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects \[[Paper](https://dl.acm.org/doi/10.1109/SC41406.2024.00039)] \[[Benchmark](https://zenodo.org/records/13312325)]
  * Sapienza University of Rome & University of Trento & Vrije Universiteit Amsterdam & ETH & CINECA & University of Antwerp & HPE & NVIDIA
  * Characterize three supercomputers: Alps, Leonardo, and LUMI.

## Acronyms

* LLM: Large Language Model
* MoE: Mixture-of-Experts
* DLRM: Deep Learning Recommendation Model
* PEFT: Parameter-Efficient Fine-Tuning
* MIG: Multi-Instance GPU
* MPS: Multi-Process Service
* CXL: Compute Express Link


# SOSP 2024

## Meta Info

Homepage: <https://sigops.org/s/conferences/sosp/2024/>

### Paper List

* <https://dl.acm.org/doi/proceedings/10.1145/3694715>
* <https://sigops.org/s/conferences/sosp/2024/accepted.html>

### Acceptance Rate

17.3% (= 43 / 248)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695960)] \[[Slides](https://swapnilgandhi.com/slides/recycle-sosp24.pdf)] \[[arXiv](https://arxiv.org/abs/2405.14009)]
    * Stanford
    * Dynamically re-route the work of a failed server to data-parallel peers; execute within bubbles of the original pipeline schedule.
  * Enabling Parallelism Hot Switching for Efficient Training of Large Language Models \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695969)] \[[Code](https://github.com/PKU-DAIR/Hetu)]
    * PKU
    * **HotSPa** — a system that adopts multiple parallelism strategies for efficient training with sequence inputs
      * Classify a mini-batch of training samples into several groups and train each group with the most suitable parallelism strategy.
      * The graph compiler generates multiple executable computation graphs that share the same backbone storage of model states.
      * Deduces efficient many-to-many communication plans of parallelism hot switching.
  * Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695975)] \[[Code](https://github.com/kungfu-team/tenplex)] \[[arXiv](https://arxiv.org/abs/2312.05181)]
    * ICL & Aalto University & Edinburgh
    * **Tenplex** — a state management library.
      * Enable jobs to change the parallelism dynamically.
      * PTC: Parallelizable Tensor Collection
        * Dataset state
        * Modle state
      * Execute PTC transformations in parallel with minimum data movement between workers.
  * Reducing Energy Bloat in Large Model Training \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695970)] \[[Code](https://github.com/ml-energy/zeus)] \[[arXiv](https://arxiv.org/abs/2312.06902)]
    * UMich
    * **Perseus**: use a graph cut-based algorithm to obtain the "iteration time-energy" Pareto frontier; schedule the energy consumption across time.
* LLM Inference
  * LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695948)] \[[Code](https://github.com/LoongServe/LoongServe)] \[[arXiv](https://arxiv.org/abs/2404.09526)]
    * PKU
    * **ESP**: Elastic Sequence Parallelism
    * Elastically adjust the degree of parallelism in real-time; reduce key-value cache migration overhead and overlap partial decoding communication with computation; reduce key-value cache fragmentation across instances.
  * PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695964)] \[[Code](https://github.com/SJTU-IPADS/PowerInfer)] \[[arXiv](https://arxiv.org/abs/2312.12456)]
    * SJTU IPADS
    * GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU.
    * Integrate adaptive predictors and neuron-aware sparse operators.

### Model Serving

* Improving DNN Inference Throughput using Practical, Per-Input Compute Adaptation \[[Paper](https://dl.acm.org/doi/pdf/10.1145/3694715.3695978)]
  * GaTech & Princeton & Stanford
  * $$E^3$$ (**E**fficient **E**arly-**E**xits): trade-off accuracy and resource costs on a per-input granularity; early-exit models.
  * Key insight: split and replicate blocks of layers in models; maintain a constant batch size throughout execution.
* Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695963)] \[[Code](https://github.com/dywsjtu/apparate)] \[[arXiv](https://arxiv.org/abs/2312.05385)]
  * Princeton & GaTech
  * Automatically apply and manage early exits (certain inputs can exit with results at intermediate layers) in ML models.

### ML Compilation

* Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695955)] \[[arXiv](https://arxiv.org/abs/2408.04808)]
  * UIUC & MSRA
  * **T10**, the first DL compiler to exploit the inter-core communication bandwidth and distributed on-chip memory on AI chips (i.e., Graphcore IPU).
* SilvanForge: A Schedule-Guided Retargetable Compiler for Decision Tree Inference \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695958)]
  * Indian Institute of Science & MSR
  * Search over several optimization choices and automatically generate high-performance inference routines for CPUs and GPUs.
  * Two components: (1) a scheduling language that encapsulates the optimization space; (2) an optimizing retargetable compiler that can generate code for any specified schedule.

### Serverless Computing

* Dirigent: Lightweight Serverless Orchestration \[[Paper](https://dl.acm.org/doi/10.1145/3694715.3695966)] \[[Code](https://github.com/eth-easl/dirigent)] \[[arXiv](https://arxiv.org/abs/2404.16393)]
  * ETH
  * Simplify state management of the existing orchestration system (Kubernetes); eliminate persistent state updates; run monolithic control and data planes to minimize internal communication overheads.


# VLDB 2024

## Meta Info

Homepage: <https://vldb.org/2024/>

### Paper list

* <https://vldb.org/2024/?program-schedule>
* <https://www.vldb.org/pvldb/volumes/17/>

## Papers

### Resource Management

* DL training workloads
  * Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/Saturn%3A%20An%20Optimized%20Data%20System%20for%20Multi-Large-Model%20Deep%20Learning%20Workloads)] \[[Code](https://github.com/knagrecha/saturn)]
    * UCSD
    * **Saturn** -> SPASE: Select a Parallelism, Allocate resources, and Schedule; formulate the joint SPASE problem as an MILP.
* Big data analytic workloads
  * Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/Intelligent%20Pooling%3A%20Proactive%20Resource%20Provisioning%20in%20Large-scale%20Cloud%20Service)]
    * Microsoft
    * Predict usage patterns using a hybrid ML model; optimize the pool size dynamically.
* Job scheduling
  * ResLake: Towards Minimum Job Latency and Balanced Resource Utilization in Geo-distributed Job Scheduling
    * ByteDance
* Autoscaling
  * OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud \[[arXiv](https://arxiv.org/abs/2311.12864)]
    * Ant Group
* Serverless
  * Resource Management in Aurora Serverless \[[Paper](https://www.amazon.science/publications/resource-management-in-aurora-serverless)]
    * AWS
    * Industry Paper

### Model Serving

* Approximate Inference
  * Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines \[[Paper](https://dl.acm.org/doi/abs/10.14778/3675034.3675052)] \[[arXiv](https://arxiv.org/abs/2405.11191)] \[[Code](https://github.com/ChaokunChang/Biathlon)]
    * CUHK
      * *Approximate* input features to accelerate inference pipelines.
      * Trade-off between latency and accuracy.
      * Evaluation: All inference pipelines were implemented using Python and scikit-learn; run on CPU servers.
  * InferDB: In-Database Machine Learning Inference Using Indexes \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/InferDB%3A%20In-Database%20Machine%20Learning%20Inference%20Using%20Indexes)]
    * Hasso Plattner Institute & University of Potsdam & University of Illinois Chicago
    * Approximate ML inference pipelines using index structures available in DBMS.
    * Predictions are preserved in the embedding space; select binned features for indexing.
    * IMO: Aggressive...
* Edge Computing
  * SmartLite: A DBMS-based Serving System for DNN Inference in Resource-constrained Environments \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/SmartLite%3A%20A%20DBMS-based%20Serving%20System%20for%20DNN%20Inference%20in%20Resource-constrained%20Environments)] \[[Code](https://github.com/lynn2089/SmartLite)]
    * ZJU & Alibaba
    * SmartLite, a lightweight DBMS
      * Store the parameters and structural information of neural networks as database tables.
      * Implement neural network operators inside the DBMS engine.
      * Quantize model parameters as binarized values, apply neural pruning techniques to compress the models, and transform tensor manipulations into value lookup operations of the DBMS.

### Notebook

* ElasticNotebook: Enabling Live Migration for Computational Notebooks \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/ElasticNotebook%3A%20Enabling%20Live%20Migration%20for%20Computational%20Notebooks)] \[[arXiv](https://arxiv.org/abs/2309.11083)] \[[Code](https://github.com/illinoisdata/ElasticNotebook)]
  * UIUC & UMich
  * Live migration via checkpointing/restoration.
  * Reconstruct all variables from a subset of variables.

### Feature Stores

* RALF: Accuracy-Aware Scheduling for Feature Store Maintenance \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/RALF%3A%20Accuracy-Aware%20Scheduling%20for%20Feature%20Store%20Maintenance)]
  * UC Berkeley
  * Limitations of existing works
    * Naively apply a one-size-fits-all policy as to when/how to update these features.
    * Do not consider query access patterns or impacts on prediction accuracy.
  * *Feature store regret*: a metric for how much featurization degrades downstream accuracy.
  * Leverage *downstream error feedback* to minimize feature store regret.

### Data Pre-processing

* FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/FusionFlow%3A%20Accelerating%20Data%20Preprocessing%20for%20Machine%20Learning%20with%20CPU-GPU%20Cooperation)] \[[Code](https://github.com/omnia-unist/FusionFlow)]
  * UNIST
  * Cooperatively utilizes *both CPUs and GPUs* to accelerate the data preprocessing stage of DL training that runs the data augmentation algorithm.
  * Orchestrate data preprocessing tasks across CPUs and GPUs while minimizing *interference* with GPU-based model training.

### Deep Learning Recommendation Model (DLRM)

* DLRover: Resource Optimization for Deep Recommendation Models Training at AntGroup \[[arXiv](https://arxiv.org/abs/2304.01468)] \[[Code](https://github.com/intelligent-machine-learning/dlrover)]
  * AntGroup & Sichuan University

### Graph Neural Network (GNN)

* Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses \[[Paper](https://www.vldb.org/pvldb/volumes/17/paper/Accelerating%20Sampling%20and%20Aggregation%20Operations%20in%20GNN%20Frameworks%20with%20GPU%20Initiated%20Direct%20Storage%20Accesses)] \[[Code](https://github.com/jeongminpark417/GIDS)]
  * UIUC & NVIDIA
  * GIDS: GPU Initiated Direct Storage Access -> A data loader to utilize all hardware resources (i.e., CPU memory, storage, and GPU memory)


# SIGCOMM 2024

## Meta Info

Homepage: <https://conferences.sigcomm.org/sigcomm/2024/>

### Paper list

* <https://conferences.sigcomm.org/sigcomm/2024/program/>
* <https://dl.acm.org/doi/proceedings/10.1145/3651890>

## Papers

### Large Language Models (LLMs)

* Systems/Networking for LLM
  * CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672274)] \[[arXiv](https://arxiv.org/abs/2310.07240)] \[[Code](https://github.com/UChi-JCL/CacheGen)] \[[Video](https://www.youtube.com/watch?v=H4_OUWvdiNo)]
    * UChicago & Microsoft & Stanford
    * **CacheGen**: A context-loading module for LLM systems.
      * Use a custom tensor encoder to encode a KV cache into more compact bitstream representations with negligible decoding overhead.
      * Adapt the compression level of different parts of a KV cache to cope with changes in available bandwidth.
    * Objective: Focus on reducing the network delay in fetching the KV cache → TTFT reduction.
  * Alibaba HPN: A Data Center Network for Large Language Model Training \[[Paper](https://doi.org/10.1145/3651890.3672265)] \[[Video](https://www.youtube.com/watch?v=s-3VLs9sd10)]
    * Alibaba Cloud
    * Experience Track
    * LLM training's characteristics
      * Produce a small number of periodic, bursty flows (e.g., 400Gbps) on each host.
      * Require GPUs to complete iterations in synchronization; more sensitive to single-point failure.
    * Alibaba High-Performance Network (**HPN**): Introduce a 2-tier, dual-plane architecture capable of interconnecting 15K GPUs within one Pod.
      * Benefits: eliminate hash polarization; simplify the optimal path selections.
  * RDMA over Ethernet for Distributed Training at Meta Scale \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672233)] \[[Blog](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/)]
    * Meta
    * Experience Track
    * Deploy a combination of centralized traffic engineering and an Enhanced ECMP (Equal-Cost Multi-Path) scheme to achieve optimal load distribution for training workloads.
    * Design a receiver-driven traffic admission via the collective library -> Co-tune both the collective library configuration and the underlying network configuration.
* LLMs for Networking
  * NetLLM: Adapting Large Language Models for Networking \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672268)]
    * CUHK-Shenzhen & Tsinghua SIGS & UChicago
    * **NetLLM**: Empower the LLM to process multimodal data in networking and generate task-specific answers.
    * Study three networking-related use cases: viewport prediction, adaptive bitrate streaming, and cluster job scheduling.

### Distributed Training

* Crux: GPU-Efficient Communication Scheduling for Deep Learning Training \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672239)] \[[Dataset](https://github.com/alibaba/alibaba-lingjun-dataset-2023)]
  * Alibaba Cloud
  * Observation: Communication contention among different deep learning training (DLT) jobs seriously influences the overall GPU computation utilization -> Low efficiency of the training cluster.
  * **Crux**: A communication scheduler
    * Objective: Mitigate the communication contention among DLT jobs -> Maximize GPU computation utilization.
    * Designs: reduce the GPU utilization problem to a flow optimization problem; GPU intensity-aware communication scheduling; prioritize the DLT flows with high GPU computation intensity.
* Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672228)]
  * KAIST & UC Irvine & VMware Research
  * **StellaTrain**: Cache-aware gradient compression; a CPU-based sparse optimizer.
  * Adapt training configurations to fluctuating dynamic network bandwidth -> Enable co-training using on-premises and cloud clusters.

### Data Processing

* Turbo: Efficient Communication Framework for Large-scale Data Processing Cluster \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672241)]
  * Tencent & FDU & NVIDIA & THU
  * Experience Track
  * Network throughput & scalability: A dynamic block-level flowlet transmission mechanism; a non-blocking communication middleware.
  * System reliability: Utilize an external shuffle service as well as TCP serving as a backup.
  * Integrated into Apache Spark.

### Data Transfers

* An exabyte a day: Throughput-oriented, Large-scale, Managed Data Transfers with Effingo \[[Paper](https://dl.acm.org/doi/10.1145/3651890.3672262)]
  * Google
  * Experience Track
  * **Effingo**: A copy system, integrated with resource management and authorization systems.
    * Per-cluster deployments -> Limit failure domains to individual clusters.
    * Separation from the bandwidth management layer (BwE) -> A modular design that reduces dependencies.


# ICML 2024

## Meta Info

Homepage: <https://icml.cc/Conferences/2024>

### Papers

### Large Language Models (LLMs)

* Serving LLMs
  * HexGen: Generative Inference of Foundation Model over Heterogeneous Decentralized Environment \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2023/hexgen)] \[[arXiv](https://arxiv.org/abs/2311.11514)] \[[Code](https://github.com/Relaxed-System-Lab/HexGen)]
    * HKUST & ETH & CMU
      * Support *asymmetric* tensor model parallelism and pipeline parallelism under the *heterogeneous* setting (i.e., each pipeline parallel stage can be assigned with a different number of layers and tensor model parallel degree).
        * Propose *a heuristic-based evolutionary algorithm* to search for the optimal layout.
  * MuxServe: Flexible Spatial-Temporal Multiplexing for LLM Serving \[[arXiv](https://arxiv.org/abs/2404.02015)] \[[Code](https://github.com/hao-ai-lab/MuxServe)]
    * CUHK & Shanghai AI Lab & HUST & SJTU & PKU & UC Berkeley & UCSD
    * Colocate LLMs considering their popularity to multiplex memory resources.
  * APIServe: Efficient API Support for Large-Language Model Inferencing \[[arXiv](https://arxiv.org/abs/2402.01869)]
    * UCSD
* Benchmark
  * Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference \[[arXiv](https://www.google.com/url?sa=t\&source=web\&rct=j\&opi=89978449\&url=https://arxiv.org/abs/2403.04132\&ved=2ahUKEwinqvnbiruHAxWZmO4BHQAfAaMQFnoECAgQAQ\&usg=AOvVaw0xl2m0cvjY2iAKescRSm3P)] \[[Demo](https://chat.lmsys.org)]
    * UC Berkeley
* Speculative decoding
  * Online Speculative Decoding \[[arXiv](https://arxiv.org/abs/2310.07177)]
    * UC Berkeley & UCSD & Sisu Data & SJTU
* Video generation
  * VideoPoet: A Large Language Model for Zero-Shot Video Generation \[[Paper](https://proceedings.mlr.press/v235/kondratyuk24a.html)] \[[Homepage](https://sites.research.google/videopoet/)]
    * Google & CMU
    * Employ a decoder-only transformer architecture that processes multimodal inputs – including images, videos, text, and audio.
    * The pre-trained LLM is adapted to a range of video generation tasks.
* Image retrieval
  * MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions \[[Paper](https://proceedings.mlr.press/v235/zhang24an.html)] \[[Homepage](https://open-vision-language.github.io/MagicLens/)] \[[Code](https://github.com/google-deepmind/magiclens)]
    * OSU & Google DeepMind
    * Enable multimodality-to-image, image-to-image, and text-to-image retrieval.

## References

* [Google DeepMind at ICML 2024, 2024/07/19](https://deepmind.google/discover/blog/google-deepmind-at-icml-2024/)


# ATC 2024

## Meta Info

Homepage: <https://www.usenix.org/conference/atc24>

Paper list: <https://www.usenix.org/conference/atc24/technical-sessions>

### Acceptance Rate

15.8% (= 77 / 488)

## Papers

### Large Language Models (LLMs)

* Serving LLMs
  * Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention \[[Paper](https://www.usenix.org/conference/atc24/presentation/gao-bin-cost)]
    * NUS & SJTU & Huawei Cloud
    * Reuse KV caches across multi-turn conversations; maintain a hierarchical KV caching system; layer-wise pre-loading and asynchronous saving; scheduler-aware fetching and eviction.
  * Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs \[[Paper](https://www.usenix.org/conference/atc24/presentation/xia)] \[[Code](https://github.com/usyd-fsalab/fp6_llm)]
    * Sydney & Microsoft & Rutgers
    * **TC-FPx**, the first full-stack *GPU kernel design* scheme with unified Tensor Core support of 6-bit and arbitrary bit-width quantization (e.g., 5-bit).
* LLM alignment / RLHF training
  * PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch \[[Paper](https://www.usenix.org/conference/atc24/presentation/lei)]
    * THU
    * Intra-stage switching: explore model affinities and overlap computation via time-sharing.
    * Inter-stage switching: find the optimal switch plan with the minimum communication cost.
    * Based on Megatron-LM.
* LLM federated fine-tuning
  * FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences \[[Paper](https://www.usenix.org/conference/atc24/presentation/xu-mengwei)] \[[Code](https://github.com/UbiquitousLearning/FwdLLM)]
    * BUPT
    * Employ backpropagation (BP)-free training methods, requiring devices only to execute “perturbed inferences”; adaptively allocate computational loads across devices to balance between convergence speed and accuracy.
* LLM training
  * Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Parallelism \[[Paper](https://www.usenix.org/conference/atc24/presentation/yuan)] \[[Code](https://github.com/kwai/Megatron-Kwai/tree/atc24ae/examples/atc24)]
    * Kuaishou
    * The balance between computation and memory utilization.
    * Two activation rematerialization strategies
      * *Pipeline-parallel-aware offloading* to maximize the utilization of host memory for storing activations.
      * *Compute-memory balanced checkpointing* to balance between activation memory and computational efficiency.

### Reliability

* AI Infra
  * SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation \[[Paper](https://www.usenix.org/conference/atc24/presentation/xiong)] \[[Code](https://github.com/microsoft/superbenchmark)]
    * MSR & Microsoft
      * **Best Paper Award**
      * SuperBench, a proactive validation system for AI infrastructure that mitigates *hidden degradation* (i.e., *gray failure*) caused by hardware redundancies and enhances overall reliability.
      * A comprehensive benchmark suite to evaluate individual hardware components and represent most real AI workloads.
* HBM
  * Removing Obstacles before Breaking Through the Memory Wall: A Close Look at HBM Errors in the Field \[[Paper](https://www.usenix.org/conference/atc24/presentation/wu-ronglong)] \[[Code](https://github.com/wrl297/Calchas)]
    * Xiamen University & Huawei & Minjiang University
      * Conduct the first systematical study on HBM errors, which cover over 460 million error events collected from nineteen data centers and span over two years of deployment under a variety of services.
      * Calchas, a hierarchical failure prediction framework for HBM integrates spatial, temporal, and sensor information from various device levels to predict upcoming failures.

### Supercomputer

* Full Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer: What Can We Learn from It?
  * THU & SDU & National Supercomputer Center in Wuxi
  * A comprehensive analysis of six years’ worth of 40 TB data (comprising I/O performance data and job running information) from Sunway TaihuLight, boasting 41508 nodes.
  * **Notice**: The data is currently not available.

### Distributed Training

* Metis: Fast Automatic Distributed Training on Heterogeneous GPUs \[[Paper](https://www.usenix.org/conference/atc24/presentation/um)]
  * Samsung Research & UNIST
  * Metis, a system to automatically finds efficient parallelism plans for distributed training on *heterogeneous GPUs*.
  * Balance loads with heterogeneity-awareness; prefer data parallelism over tensor parallelism within a pipeline stage.
  * Evaluated with three large models (GPT-3, MoE, and Wide-Resnet).

### Data Preprocessing

* Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement \[[Paper](https://www.usenix.org/conference/atc24/presentation/graur)] \[[Code](https://github.com/eth-easl/pecan-experiments)]
  * ETH & Google
  * Dynamically *schedule data preprocessing workers on ML accelerator host resources* to minimize the number of remote CPU workers needed to achieve peak data ingestion bandwidth.
  * Analyze the characteristics of input pipelines and *automatically reorder transformations* to increase data preprocessing worker throughput

### Serverless Computing

* Harmonizing Efficiency and Practicability: Optimizing Resource Utilization in Serverless Computing with Jiagu \[[Paper](https://www.usenix.org/conference/atc24/presentation/liu-qingyuan)]
  * SJTU, IPAPS & Huawei Cloud & EPFL
  * **Jiagu**, a serverless system based on OpenFaaS
    * *Pre-decision scheduling:* decouple prediction and decision-making; predict every function's capacities on a server using a model.
    * *Dual-staged scaling:* frequent adjustment of instances.
* ALPS: An Adaptive Learning, Priority OS Scheduler for Serverless Functions \[[Paper](https://www.usenix.org/conference/atc24/presentation/fu)] \[[Code](https://github.com/ds2-lab/ALPS)]
  * UVA & George Mason University & Adobe Research
  * **ALPS**: **A**daptive **L**earning, **P**riority **S**cheduler
    * Application-aware kernel scheduler
    * Frontend: user-space; approximate *shortest remaining process time* (SRPT) priority scheduling by adaptively learning from an SRPT simulation on recent past workload.
    * Backend: use eBPF functions hooked to CFS to inform scheduling decisions (from the frontend) in the kernel.
* StreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow \[[Paper](https://www.usenix.org/conference/atc24/presentation/wu-hao)] \[[Code](https://github.com/CGCL-codes/streambox)]
  * HUST & INRIA
  * *One GPU runtime per inference workflow* instead of *one GPU runtime per function*.
  * Use CUDA streams for serverless inference; fine-grained GPU memory management; and PCIe bandwidth sharing among concurrent streams.
* A Secure, Fast, and Resource-Efficient Serverless Platform with Function REWIND \[[Paper](https://www.usenix.org/conference/atc24/presentation/song)] \[[Code](https://github.com/s3yonsei/rewind_serverless)]
  * Sungkyunkwan University & Yonsei University & Seoul National University
  * Enhance performance while *maintaining strict data isolation between requests*.
  * The container is reset to an initial state free of any sensitive data after each function request; incorporate a kernel-level memory snapshot management system; optimize runtime by reusing memory regions and leveraging the temporal locality of function executions.

### Model Serving

* Power-aware Deep Learning Model Serving with μ-Serve \[[Paper](https://www.usenix.org/conference/atc24/presentation/qiu)]
  * UIUC & IBM Research
  * Scaling GPU frequency for power saving without SLO attainment violations.

### Cluster Scheduler

* Starburst: A Cost-aware Scheduler for Hybrid Cloud \[[Paper](https://www.usenix.org/conference/atc24/presentation/luo)] \[[Code](https://github.com/michaelzhiluo/starburst)]
  * UC Berkeley & UCSB
  * Distinguished Artifact Award
  * Run the batch workloads on the private clusters or public cloud. -> Trade-off between the cost and JCT.
  * Dynamically control jobs' waiting times to improve utilization.
    * Assign longer waits for large jobs to increase their chances of running on the cluster.
    * Assign shorter waits to small jobs to increase their chances of running on the cloud.

### Deep Learning Compiler

* MAGPY: Compiling Eager Mode DNN Programs by Monitoring Execution States \[[Paper](https://www.usenix.org/conference/atc24/presentation/zhang-chen)] \[[Code](https://github.com/heheda12345/MagPy)]
  * THU
  * Generate more complete operator graphs by collecting key runtime information through monitoring program execution.
  * Provide a reference graph to record program execution states and leverage reference relationships to identify state changes that can impact program outputs.

### Deep Learning Recommendation Models (DLRMs)

* OPER: Optimality-Guided Embedding Table Parallelization for Large-scale Recommendation Model \[[Paper](https://www.usenix.org/conference/atc24/presentation/wang)]
  * UCSD & UCSB & Meta & Pacific Northwest National Laboratory
  * Provide a near-optimal parallelization strategy for embedding tables.

### Probabilistic Graphical Models

* Fast Inference for Probabilistic Graphical Models \[[Paper](https://www.usenix.org/conference/atc24/presentation/jiang)] \[[Code](https://github.com/jjiantong/FastPGM)]
  * University of Western Australia & HKUST
  * **Fast-PGM**: a fast and parallel PGM inference system for importance sampling-based approximate inference algorithms.

### Remote Direct Memory Access (RDMA)

* PeRF: Preemption-enabled RDMA Framework \[[Paper](https://www.usenix.org/conference/atc24/presentation/lee)]
  * Acryl Inc. & Sungkyunkwan University
  * Offer *software-based performance isolation* for efficient *multi-tenancy* in RDMA.

### Remote Procedure Call (RPC)

* HydraRPC: RPC in the CXL Era \[[Paper](https://www.usenix.org/conference/atc24/presentation/ma)]
  * Alibaba & THU & ZJU & PKU
  * Utilize CXL-attached HDM to build RPC systems.

### Journaling File System

* FastCommit: Resource-Efficient, Performant and Cost-Effective File System Journaling
  * Google
  * **Best Paper Award**

### Rust-for-Linux

* An Empirical Study of Rust-for-Linux: The Success, Dissatisfaction, and Compromise \[[Paper](https://www.usenix.org/conference/atc24/presentation/li-hongyu)] \[[Code](https://github.com/Richardhongyu/rfl_empirical_tools)]
  * BUPT & UESTC


# OSDI 2024

## Meta Info

Homepage: <https://www.usenix.org/conference/osdi24>

Paper list: <https://www.usenix.org/conference/osdi24/technical-sessions>

### Acceptance Rate

19.2% (= 53 / 276)

## Papers

### Large Language Models (LLMs)

* Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve \[[Paper](https://www.usenix.org/conference/osdi24/presentation/agrawal)] \[[Code](https://github.com/microsoft/sarathi-serve)]
  * MSR India & GaTech
  * **Sarathi-Serve**
    * Chunked-prefills: split a prefill request into *near equal-sized chunks*; create stall-free schedules that add new requests in a batch *without pausing ongoing decodes*.
    * Stall-free scheduling: improve throughput with large batch sizes; minimize the effect of batching on latency.
* ServerlessLLM: Low-Latency Serverless Inference for Large Language Models \[[Paper](https://www.usenix.org/conference/osdi24/presentation/fu)] \[[Code](https://github.com/ServerlessLLM/ServerlessLLM)]
  * Edinburgh
  * Multi-tier checkpoint loading.
  * Live migration of LLM inference: the source server migrates only the tokens; a re-computation of the KV-cache is triggered at the destination server.
  * Use cost models to estimate the time of loading checkpoints from different tiers in the storage hierarchy and the time of migrating an LLM inference to another server; choose the best server to minimize model startup latency.
* InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management \[[Paper](https://www.usenix.org/conference/osdi24/presentation/lee)]
  * Seoul National University
  * **InfiniGen**: a *KV cache management* framework for *long-text generation*.
  * Key insight: A few important tokens can be speculated by performing a minimal rehearsal with the inputs of the current layer and part of the query weight and key cache of the subsequent layer.
  * Prefetch only the essential KV cache entries instead of fetching them all. -> Mitigate the fetch overhead from the host memory.
* Llumnix: Dynamic Scheduling for Large Language Model Serving \[[Paper](https://www.usenix.org/conference/osdi24/presentation/sun-biao)] \[[Code](https://github.com/AlibabaPAI/llumnix)]
  * Alibaba
  * *Reschedule requests* to improve load-balancing and isolation, mitigate resource fragmentation, and differentiate request priorities and SLOs.
  * Live migration for requests and the in-memory states (tokens).
* DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin)] \[[Code](https://github.com/LLMServe/DistServe)]
  * PKU & UCSD
  * Disaggregate the prefill and decoding computation.
  * Co-optimize the resource allocation and parallelism strategy for each phase; consider the cluster's bandwidth to minimize the communication overhead.
* dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving \[[Paper](https://www.usenix.org/conference/osdi24/presentation/wu-bingyang)]
  * PKU & Shanghai AI Lab
  * A credit-based batching algorithm to decide when to *merge and unmerge* LoRA adapters with the base model.
  * A request-adapter co-migration algorithm to decide when to *migrate* between different worker replicas.
* Parrot: Efficient Serving of LLM-based Applications with Semantic Variable \[[Paper](https://www.usenix.org/conference/osdi24/presentation/lin-chaofan)] \[[Code](https://github.com/microsoft/ParrotServe)]
  * SJTU & MSRA
  * **Semantic Variable**: a unified abstraction to expose application-level knowledge to public LLM services.
    * Annotate an input/output variable in the prompt of a request.
    * Create the data pipeline when connecting multiple LLM requests.
    * Allow to perform conventional data flow analysis to uncover the correlation across multiple LLM requests.
  * Implemented on Python.
* Fairness in Serving Large Language Models \[[Paper](https://www.usenix.org/conference/osdi24/presentation/sheng)] \[[Code](https://github.com/Ying1123/VTC-artifact)]
  * UC Berkeley
  * This is the *first* work to discuss the *fair serving* of LLMs.
  * Propose a fair-serving algorithm called **Virtual Token Counter** (**VTC**).
    * Track the services received for each client.
    * Prioritize the ones with the least services received.
    * Only manipulate the dispatch order and don't reject a request if it can fit in the batch.

### Resource Allocation

* Optimizing Resource Allocation in Hyperscale Datacenters: Scalability, Usability, and Experiences \[[Paper](https://www.usenix.org/conference/osdi24/presentation/kumar)]
  * Meta Platforms
  * Main challenges for a resource-allocation framework.
    * Usability: how to translate real-life policies into precise mathematical formulas.
    * Scalability: NP-hard problems that cannot be solved efficiently by commercial solvers.
  * **Rebalancer**: Meta's resource-allocation framework.
    * An expression graph that enables its optimization algorithm to run more efficiently than past algorithms (for scalability).
    * A high-level specification language to lower the barrier for adoption by system practitioners (for usability).

### Job Scheduling

* When will my ML Job finish? Toward providing Completion Time Estimates through Predictability-Centric Scheduling \[[Paper](https://www.usenix.org/conference/osdi24/presentation/bin-faisal)] \[[Code](https://github.com/TuftsNATLab/PCS)]
  * Tufts
  * PCS: Predictability-Centric Scheduling
  * Use Weighted-Fair-Queueing (WFQ) and find a suitable configuration of different WFQ parameters (e.g., queue weights).
  * Use a simulation-aided search strategy to discover WFQ configurations.
* MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale \[[Paper](https://www.usenix.org/conference/osdi24/presentation/choudhury)]
  * Meta Platforms
  * MAST: ML Application Scheduler on Twine
  * Provide a global-scheduling abstraction to all ML training workloads.
  * Three design principles: temporal decoupling, scope decoupling, and exhaustive search.

### Auto Parallelization

* nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training \[[Paper](https://www.usenix.org/conference/osdi24/presentation/lin-zhiqi)] \[[Code](https://github.com/microsoft/nnscaler)]
  * USTC & MSRA & xAI & BaseBit Technologies
  * Empower domain experts to construct their own search space through three primitives, `op-trans`, `op-assign`, and `op-order`.
  * Allow the application of constraints to those primitives during space construction.

### Machine Learning Inference

* Usher: Holistic Interference Avoidance for Resource Optimized ML Inference \[[Paper](https://www.usenix.org/conference/osdi24/presentation/shubha)] \[[Code](https://github.com/ss7krd/Usher)]
  * UVA & GaTech
  * Usher: an interference-aware ML serving system to maximize resource utilization (GPU spatial multiplexing).
    * GPU kernel-based model resource requirement estimator.
    * Heuristic-based interference-aware resource utilization-maximizing scheduler that decides the batch size, model replication degree, and model placement.
    * Operator graph merger to merge multiple models to minimize interference in GPU cache.

### Tensor Program Generation

* Enabling Tensor Language Model to Assist in Generating High-Performance Tensor Programs for Deep Learning \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhai)] \[[Code](https://github.com/zhaiyi000/tlm)]
  * USTC & Huawei & ByteDance & Hunan University
  * Tensor Language Model (TLM)
* Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transformation \[[Paper](https://www.usenix.org/conference/osdi24/presentation/wang-lei)] \[[Code](https://github.com/microsoft/BitBLAS/tree/osdi24_ladder_artifact)]
  * MSRA
* MonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference Tasks on Modern GPU-Centric Architectures \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhuang)] \[[Code](https://github.com/AlibabaResearch/mononn)]
  * Sydney & Alibaba
  * The code is currently not available.

### Machine Learning APIs

* ChameleonAPI: Automatic and Efficient Customization of Neural Networks for ML Applications \[[Paper](https://www.usenix.org/conference/osdi24/presentation/liu)] \[[Code](https://github.com/UChi-JCL/chameleonAPI)]
  * UChicago & ECNU & MSR

### In-Network Machine Learning

* Caravan: Practical Online Learning of In-Network ML Models with Labeling Agents \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhang-qizheng)] \[[Code](https://github.com/Per-Packet-AI/Caravan-Artifact-OSDI24)]
  * Stanford & Princeton & Sapienza University of Rome & UMich

### Microkernel

* Microkernel Goes General: Performance and Compatibility in the HongMeng Production Microkernel \[[Paper](https://www.usenix.org/conference/osdi24/presentation/chen-haibo)]
  * Huawei Central Software Institute & SJTU
  * Hong-Meng kernel (HM)

### Compute Express Link (CXL)

* Managing Memory Tiers with CXL in Virtualized Environments \[[Paper](https://www.usenix.org/conference/osdi24/presentation/zhong-yuhong)]
  * Columbia & Microsoft Azure & UW & Carl Waldspurger Consulting & Intel & UW-Madison & UMich

### Distributed Snapshots

* Beaver: Practical Partial Snapshots for Distributed Cloud Services \[[Paper](https://www.usenix.org/conference/osdi24/presentation/yu)] \[[Code](https://github.com/eniac/Beaver)]
  * UPenn & SJTU & Princeton & Microsoft & UW

### Network Interface Card (NIC)

* High-throughput and Flexible Host Networking for Accelerated Computing \[[Paper](https://www.usenix.org/conference/osdi24/presentation/skiadopoulos)] \[[Code](https://github.com/enfabrica/iperf)]
  * Stanford & Cornell & Enfabrica

### Collective Communication Library

* ACCL+: an FPGA-Based Collective Engine for Distributed Applications \[[Paper](https://www.usenix.org/conference/osdi24/presentation/he)]
  * ETH & Amsterdam & AMD

### Hardware Accelerators

* Performance Interfaces for Hardware Accelerators \[[Paper](https://www.usenix.org/conference/osdi24/presentation/ma-jiacheng)] \[[Code](https://github.com/dslab-epfl/lpn)]
  * EPFL
  * LPN: Latency Petri Net

### Cloud Block Storage <a href="#page-title" id="page-title"></a>

* Burstable Cloud Block Storage with Data Processing Units \[[Paper](https://www.usenix.org/conference/osdi24/presentation/shu)]
  * PKU & Alibaba Cloud

### Formal Verification

* Anvil: Verifying Liveness of Cluster Management Controllers \[[Paper](https://www.usenix.org/conference/osdi24/presentation/sun-xudong)] \[[Code](https://github.com/vmware-research/verifiable-controllers)]
  * UIUC & UW-Madison & VMware Research & Feldera
  * **Best Paper Award**

## References

* Notes from SJTU IPADS (in Chinese)
  * [OSDI 2024 论文评述 Day 1 Session 1: Memory Management - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/707983034)
  * [OSDI 2024 论文评述 Day 1 Session 2: Low-Latency LLM Serving - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/707990822)
  * [OSDI 2024 论文评述 Day 1 Session 3: Distributed Systems - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/707998884)
  * [OSDI 2024 论文评述 Day 2 Session 4: Deep Learning - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708002201)
  * [OSDI 2024 论文评述 Day 2 Session 5: Operating Systems - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708003676)
  * [OSDI 2024 论文评述 Day 2 Session 6: Cloud Computing - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708034284)
  * [OSDI 2024 论文评述 Day 2 Session 7: Formal Verification - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708035509)
  * [OSDI 2024 论文评述 Day 3 Session 8: Cloud Security - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708036283)
  * [OSDI 2024 论文评述 Day 3 Session 9: Data Management - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708037149)
  * [OSDI 2024 论文评述 Day 3 Session 10: Analysis of Correctness - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708037498)
  * [OSDI 2024 论文评述 Day 3 Session 11: ML Scheduling - IPADS-SYS 的文章 - 知乎](https://zhuanlan.zhihu.com/p/708038262)


# ISCA 2024

## Meta Info

Homepage: <https://iscaconf.org/isca2024/>

Paper list: <https://www.iscaconf.org/isca2024/program/>

## Papers

### Large Language Models (LLMs)

* Splitwise: Efficient Generative LLM Inference Using Phase Splitting
  * Microsoft
  * **Best Paper Award**
* MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix Partition
* Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization
* LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
* ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching

### Mixture-of-Experts (MoEs)

* Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  * MSRA
  * Pre-gating function to alleviate the dynamic nature of sparse expert activation. -> Address the large memory footprint.

### Recommendation Models

* Heterogeneous Acceleration Pipeline for Recommendation System Training \[[arXiv](https://arxiv.org/abs/2204.05436)]
  * UBC & GaTech
  * **Hotline**: a runtime framework.
  * Utilize CPU main memory for non-popular embeddings and GPUs’ HBM for popular embeddings.
  * Fragment a mini-batch into popular and non-popular micro-batches (μ-batches).

### Diffusion Models

* Cambricon-D: Full-Network Differential Acceleration for Diffusion Models
  * ICT, CAS
  * The first processor design to address Diffusion Model acceleration.
  * Mitigate additional memory accesses, while maintaining the concise computation from differential computing.

### Video Analytics

* DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics

### Accelerators

* Intel Accelerator Ecosystem: An SoC-Oriented Perspective
  * Intel
  * Industry Session


# CVPR 2024

## Meta Info

Homepage: <https://cvpr.thecvf.com/Conferences/2024>

Paper list: <https://cvpr.thecvf.com/Conferences/2024/AcceptedPapers>

## Papers

### Diffusion Models

#### Acceleration

* Cache Me if You Can: Accelerating Diffusion Models through Block Caching \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Wimbauer_Cache_Me_if_You_Can_Accelerating_Diffusion_Models_through_Block_CVPR_2024_paper.html)] \[[Homepage](https://fwmb.github.io/blockcaching/)]
  * Meta & TUM & MCML & Oxford
  * **Block caching**
    * Reuse outputs from layer blocks of previous steps to speed up inference.
    * Automatically determines caching schedules based on each block's changes over timesteps.
* CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Zeng_CAT-DM_Controllable_Accelerated_Virtual_Try-on_with_Diffusion_Model_CVPR_2024_paper.html)] \[[Code](https://github.com/zengjianhao/CAT-DM)]
  * TJU & Tencent
  * **CAT-DM**: **C**ontrollable **A**ccelerated virtual **T**ry-on with **D**iffusion **M**odel
  * Initiate a reverse denoising process with an implicit distribution generated by a pre-trained GAN-based model → Reduce the sampling steps
* DeepCache: Accelerating Diffusion Models for Free \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Ma_DeepCache_Accelerating_Diffusion_Models_for_Free_CVPR_2024_paper.html)] \[[Code](https://github.com/horseee/DeepCache)]
  * NUS
  * Utilize the inherent temporal redundancy observed in the sequential denoising steps of diffusion models.
  * Cache and retrieve features across adjacent denoising stages, thereby reducing redundant computations.
* DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Li_DistriFusion_Distributed_Parallel_Inference_for_High-Resolution_Diffusion_Models_CVPR_2024_paper.html)] \[[Homepage](https://hanlab.mit.edu/projects/distrifusion)] \[[Code](https://github.com/mit-han-lab/distrifuser)]
  * MIT & Princeton & Lepton AI & NVIDIA
  * Displaced patch parallelism
    * Split the model input into multiple patches and assign each patch to a GPU.
    * Reuse the pre-computed feature maps from the previous timestep to provide context for the current step.
  * **DistriFusion** → Enable running diffusion models across multiple GPUs in parallel
* SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Nguyen_SwiftBrush_One-Step_Text-to-Image_Diffusion_Model_with_Variational_Score_Distillation_CVPR_2024_paper.html)]
  * VinAI Research, Vietnam
  * **Knowledge distillation:** Distill a pre-trained multi-step text-to-image model to a student network that can generate images with *just a single inference step*.

#### Support compatibility of add-on modules (ControlNets and LoRAs)

* X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Ran_X-Adapter_Adding_Universal_Compatibility_of_Plugins_for_Upgraded_Diffusion_Model_CVPR_2024_paper.html)] \[[Homepage](https://showlab.github.io/X-Adapter/)] \[[Code](https://github.com/showlab/X-Adapter)]
  * NUS & Tencent & FDU
  * Enable the pre-trained add-on modules (ControlNet, LoRA) with the upgraded diffusion model (SDXL) *without further retraining*.

#### Support arbitrary image size

* ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Haji-Ali_ElasticDiffusion_Training-free_Arbitrary_Size_Image_Generation_through_Global-Local_Content_Separation_CVPR_2024_paper.html)] \[[Homepage](https://elasticdiffusion.github.io)] \[[Code](https://github.com/MoayedHajiAli/ElasticDiffusion-official)]
  * Rice University
  * Enable pre-trained text-to-image diffusion models to generate images with various sizes.
  * Decouple the generation trajectory of a pre-trained model into local and global signals.
    * The local signal controls low-level pixel information and can be estimated on local patches.
    * The global signal is used to maintain overall structural consistency and is estimated with a reference image.

#### Improve image quality

* FreeU: Free Lunch in Diffusion U-Net \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Si_FreeU_Free_Lunch_in_Diffusion_U-Net_CVPR_2024_paper.html)] \[[Homepage](https://chenyangsi.top/FreeU/)] \[[Code](https://github.com/ChenyangSi/FreeU)]
  * NTU
  * Key insight
    * Use two modulation factors to re-weight the feature contributions from the U-Net’s skip connections and backbone.
    * Increasing the backbone scaling factor *b* significantly enhances image quality.
    * Directly scaling *s* in the skip features has a limited influence on image synthesis quality.
  * **FreeU**
    * Improve the generation quality with only a few lines of code.
    * Only need to adjust two scaling factors during the inference.

#### Scalability

* On the Scalability of Diffusion-based Text-to-Image Generation \[[Paper](https://openaccess.thecvf.com/content/CVPR2024/html/Li_On_the_Scalability_of_Diffusion-based_Text-to-Image_Generation_CVPR_2024_paper.html)]
  * AWS AI Labs & Amazon AGI
  * An empirical study of the scaling properties of diffusion-based text-to-image models.
  * Perform ablations on scaling both denoising backbones and training set, including training scaled U-Net and Transformer variants ranging from 0.4B to 4B parameters on datasets up to 600M images.
  * Specifically
    * Model scaling
      * The location and amount of cross-attention distinguish the performance.
      * To improve text-image alignment, increasing the transformer blocks is more parameter-efficient than increasing channel numbers.
      * Identify an efficient UNet variant.
    * Data scaling
      * The quality and diversity of the training set matter more than simply dataset size.
    * Provide scaling functions to predict the text-image alignment performance as functions of the scale of model size, compute, and dataset size.


# MLSys 2024

## Meta Info

Homepage: <https://mlsys.org/Conferences/2024>

Paper list: <https://mlsys.org/Conferences/2024/AcceptedPapers>

## Papers

### Large Language Models (LLMs)

* LoRA serving
  * S-LoRA: Serving Thousands of Concurrent LoRA Adapters \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/906419cd502575b617cc489a1a696a67-Paper-Conference.pdf)] \[[arXiv](https://arxiv.org/abs/2311.03285)] \[[Code](https://github.com/S-LoRA/S-LoRA)]
    * UC Berkeley
      * A system to serve many LoRA adapters
      * Store all adapters in the main memory and fetch the adapters used by the currently running queries to the GPU memory
      * Unified Paging — a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths
      * Employ a tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation
      * Built on top of [LightLLM](https://github.com/ModelTC/lightllm)
  * Punica: Multi-Tenant LoRA Serving \[[arXiv](https://arxiv.org/abs/2310.18547)] \[[Code](https://github.com/punica-ai/punica)]
    * UW & Duke
      * A system to serve multiple LoRA models in a shared GPU cluster
      * A CUDA kernel — Segmented Gather Matrix-Vector Multiplication (SGMV)
        * Batch GPU operations for concurrent execution of different LoRA models
        * A GPU only needs to store a single copy of the pre-trained model
      * A request scheduling mechanism to consolidate multi-tenant LoRA serving workloads
        * Route the new request to a small set of active GPUs
        * Allocate additional GPU resources when the existing GPUs are fully utilized
        * Periodically migrate existing requests for consolidation
* LLM inference
  * Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/48fecef47b19fe501d27d338b6d52582-Paper-Conference.pdf)] \[[Code](https://github.com/d-matrix-ai/keyformer-llm)]
    * UBC & d-Matrix
  * Prompt Cache: Modular Attention Reuse for Low-Latency Inference \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/a66caa1703fe34705a4368c3014c1966-Paper-Conference.pdf)] \[[arXiv](https://arxiv.org/abs/2311.04934)]
    * Yale & Google
  * HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/5431dca75a8d2abc1fb51e89e8324f10-Paper-Conference.pdf)]
    * NUS
  * Vidur: A Large-scale Simulation Framework for LLM Inference \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/b74a8de47d2b3c928360e0a011f48351-Paper-Conference.pdf)] \[[Code](https://github.com/microsoft/vidur)]
    * GaTech & MSR India
  * FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/5321b1dabcd2be188d796c21b733e8c7-Paper-Conference.pdf)]
    * THU & Infinigence-AI
* LLM fine-tuning
  * Fine-Tuning Language Models Using Formal Methods Feedback: A Use Case in Autonomous Systems \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/b0131b6ee02a00b03fc3320176fec8f5-Paper-Conference.pdf)]
    * UT-Austin
* LLM for data manipulation
  * UniDM: A Unified Framework for Data Manipulation with Large Language Models \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/dcb38c6ad7911842ab31081be9540b89-Paper-Conference.pdf)]
    * Alibaba & USTC

### Mixture-of-Experts (MoEs)

* MoE training
  * Lancet: Accelerating Mixture-of-Experts Training by Overlapping Weight Gradient Computation and All-to-All Communication \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/339caf45a6fa281cae8adc6465343464-Paper-Conference.pdf)]
    * HKU & AWS & Boson AI
* MoE inference
  * QMoE: Sub-1-Bit Compression of Trillion Parameter Models \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/c74b624843218d9b6713fcf299d6d5e4-Paper-Conference.pdf)] \[[Code](https://github.com/IST-DASLab/qmoe)]
    * Institute of Science and Technology Austria
  * SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

### Diffusion Models

* DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/45c1f6a8cbf2da59ebf2c802b4f742cd-Paper-Conference.pdf)]
  * HKU & AWS

### Deep Learning Recommendation Models (DLRMs)

* Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large Scale Recommendation \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/78834433edc3291f4c6cbbd2759324db-Paper-Conference.pdf)]
  * Meta AI

### ML Compilation

* ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/096b1019463f34eb241e87cfce8dfe16-Paper-Conference.pdf)]
  * CMU
  * Perform hybrid static+dynamic compiler optimizations and end-to-end tensor code generation

### Quantization

* FP8
  * Efficient Post-training Quantization with FP8 Formats \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/dea9b4b6f55ae611c54065d6fc750755-Paper-Conference.pdf)]
    * Intel
* LLM
  * AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf)] \[[Code](https://github.com/mit-han-lab/llm-awq)]
    * MIT
    * **Best Paper Award**
  * Atom: Low-bit Quantization for Efficient and Accurate LLM Serving \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/5edb57c05c81d04beb716ef1d542fe9e-Paper-Conference.pdf)] \[[Code](https://github.com/efeslab/Atom)] \[[Slides](https://github.com/efeslab/Atom/blob/main/figures/atom_mlsys_slides.pdf)] \[[Poster](https://github.com/efeslab/Atom/blob/main/figures/atom_mlsys_poster.pdf)]
    * UW
  * Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/bbb7506579431a85861a05fff048d3e1-Paper-Conference.pdf)] \[[Code](https://github.com/VITA-Group/Q-Hitter)]
    * UT-Texas & Oxford & Eindhoven University of Technology & Lawrence Livermore National Laboratory & CMU
* ML training
  * JIT-Q: Just-in-time Quantization with Processing-In-Memory for Efficient ML Training \[[Paper](https://arxiv.org/pdf/2311.05034)] \[[Slides](https://mlsys.org/media/mlsys-2024/Slides/2660.pdf)]
    * AMD

### Model Adaptation

* FLASH: Fast Model Adaptation in ML-Centric Cloud Platforms \[Paper] \[[Code](https://gitlab.engr.illinois.edu/DEPEND/flash)] \[[Slides](https://haoran-qiu.com/slides/flash-slides.pdf)]

### Cloud Configuration Generation

* CloudEval-YAML: A Practical Benchmark for Cloud Native YAML Configuration Generation \[[Paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/554e056fe2b6d9fd27ffcd3367ae1267-Paper-Conference.pdf)] \[[Homepage](https://cloudeval-yaml.github.io)] \[[Code](https://github.com/alibaba/CloudEval-YAML)] \[[Benchmark](https://huggingface.co/datasets/ai4cloud/CloudEval-YAML)]
  * Alibaba Cloud & UMich & UCLA & UC Merced

## Acronyms

* ML: Machine Learning
* LLM: Large Language Model
* LoRA: Low-Rank Adaptation
* MoE: Mixture-of-Experts


# ASPLOS 2024

## Meta Info

Homepage: <https://asplos-conference.org/2024/>

## Papers

### LLM Inference

* SpotServe: Serving Generative Large Language Models on Preemptible Instances \[[Personal Notes](/reading-notes/conference/asplos-2024/spotserve)] \[[Paper](https://arxiv.org/abs/2311.15566)] \[[Code](https://github.com/Hsword/SpotServe)]
  * CMU & PKU & CUHK
  * Distributed LLM serving system on preemptible/spot instances
  * Techniques
    * Dynamically adapt the LLM parallelization configuration
    * Minimize the cost of migrating instances for dynamic re-parallelization
    * Stateful inference recovery

### Model Serving

* Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling \[[Paper](https://doi.org/10.1145/3617232.3624849)]
  * UMass-Amherst & Nokia Bell Labs

### Elastic Training

* Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters
  * UMacau


# SpotServe: Serving generative large language models on preemptible instances

## Meta Info

Presented in [ASPLOS 2024](https://arxiv.org/abs/2311.15566).

## Understanding the paper

### TL;DR

* SpotServe — the first distributed LLM serving system on *preemptible instances*
* Techniques
  * Dynamically adapt the LLM parallelization configuration
  * Minimize the cost of migrating instances for dynamic reparallelization
    * Formulated as a bipartite graph matching problem → use the KuhnMunkres algorithm to identify an optimal migration plan
  * Stateful inference recovery
    * Commit inference progress at a much finer granularity
    * Resume inference upon preemption

### Background

* Spot instances
  * Lower price than on-demand instances
  * May be preempted at any time
  * Grace period (e.g., 30 seconds for AWS spot instances)

### Existing works

* Leverage spot instances to reduce the monetary cost of DNN inference
  * Example: [MArk](https://www.usenix.org/conference/atc19/presentation/zhang-chengliang), [Cocktail](https://www.usenix.org/conference/nsdi22/presentation/gunasekaran)
  * Limitation: Target *small* DNN models that can *fit on a single spot instance* with one or multiple GPU
  * Handle preemptions using *request rerouting* or *redundant computation*

### Challenges in serving LLMs on spot GPU instances

* Dynamic reparallelization — how to quickly adapt to changes to spot instances’ availability and requests’ arrival rates?
* Instance migration — how to minimize the cost of migrating GPU instances for reparallelization?
* Grace period — how to leverage grace period to handle unfinished request?

### Designs

<figure><img src="https://819228986-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkzeiawY8SkBarQBDVm-659326392%2Fuploads%2FSkmd53GfRmVyCbf5SAku%2Fimage.png?alt=media&amp;token=f997973a-eff9-4b1b-9519-05f7c266254e" alt=""><figcaption><p>An overview of SpotServe</p></figcaption></figure>

* **Inference Server**
  * Deployed on a dedicated *on-demand CPU instance*
  * Three components
    * **Request Manager**
      * Receiving input requests
      * Dynamically partition them into batches
      * Assign these batches to inference instances running on spot GPU instances
      * Collect generated outputs from the inference instances
      * Send the results back to users
    * **Meta-context Manager**
      * Manage the adjustment of the parallel configuration by sending instructions for context migration to all GPU instances
      * Modules
        * **Parallelization Controller**
          * Adjust the parallelization configuration to improve LLM serving performance
          * A parallel configuration — $$C = (D, P, M, B)$$
            * $$D$$ — data parallelism degree
            * $$P$$ — pipeline-model parallelism degree
            * $$M$$ — tensor-model parallelism degree
            * $$B$$ — the maximum mini-batch size
          * Measure the initialization time in advance
          * *Adaptive optimization algorithm*
            * Two variables $$C\_t$$ and at time step $$t$$
              * $$C\_t$$ — parallel configuration
              * $$N\_t$$ — the number of available instances
                * Include newly allocated instances
                * Exclude instances to be preempted
            * Minimizes the end-to-end inference latency $$l\_{req}(C)$$ while maintaining a throughput higher than $$\alpha\_𝑡$$
            * If multiple configurations can achieve similar minimum inference latency → Select the configuration with *lower monetary cost* (i.e., using fewer instances)
            * If peak serving throughput can not exceed the request arrival rate $$\alpha\_𝑡$$ → Maximize the overall serving throughput
            * *Optionally* allocate on-demand instances to improve serving throughput
            * Run the online algorithm, negligible overhead (i.e., less than 1s)
            * Offline estimate the latency of different configurations in advance
        * **Device Mapper**
          * Use the Kuhn-Munkres (KM) algorithm to find an optimal device mapping → Maximally reuse the model parameters and KV cache on available GPU instances & minimize the total data transmission
          * Device mapping — a bipartite graph $$\mathcal{G} = (\mathcal{V\_a}, \mathcal{V\_t}, \varepsilon)$$
            * $$u \in \mathcal{V\_a}$$ — a GPU device
            * $$v \in \mathcal{V\_t}$$ — a pipeline-stage-shard position of the parallel configuration
            * a weighted edge $$𝑒\_{𝑢𝑣}$$ — the amount of reusable model parameters and key/value cache when mapping GPU 𝑢 to position 𝑣 of the parallel configuration
          * Build a complete bipartite graph and compute the edge weight between every $$(𝑢, 𝑣)$$ pair using the size of their intersection contexts
          * If the new parallel configuration handles less concurrent inference requests
            * Discard part of the cached results → avoid exceeding the memory capacity of the new parallel configuration
            * Keep the batches of requests with more decoding progresses
        * **Migration Planner**
          * Determine the exact *migration plan* to finish the configuration adjustment
          * *Progressive migration* schedule — utilize the pipeline structure and prioritize the migration of front model layers’ context
          * Consider the memory usage during the progressive migration process
    * **Instance Manager**
      * Interacts with the cloud and receives instance preemption/acquisition notifications
      * *Allocates on-demand and spot instances at the same time* to avoid the waiting overhead when spot-instance allocation fails
      * Prefer to release on-demand instances
      * *Keep few additional instances* (e.g., 2 in experiments) to alleviate the impacts of frequent disturbance of instance availability
* **Inference Engine**
  * Deployed on each spot or on-demand GPU instance to serve LLM inference
  * Components
    * **Context daemon**
      * Manages the model parameters (i.e., *model context*) and intermediate activations (i.e., *cache context*) for different requests inside a certain GPU
    * **Interruption Arranger**
      * Support stateful inference recovery
* Stateful inference recovery
  * Recover interrupted inference request without recomputation
  * Context daemon maintains the cache context of an inference request
  * Route the request to another inference pipeline using the cached state
  * Just-in-time arrangement
    * Each spot GPU instance includes *an interruption arranger* that receives a notification when a grace period starts
  * Fault tolerance
    * Delay the acquired instance joining and make the arrangements for prior interruptions feasible
    * One instance gets preempted before expected → Give up the cache context and only migrate the model context with the rest instances
    * All replicas of the same piece of model context are lost due to unexpected failures → Restart by loading weights locally (e.g., disk) or from remote cloud storage (e.g., S3) to fetch the required model parameters

### Implementation

* 5.6K LoC in C++ and 2.2 LoC in Python
* Built on top of [FasterTransformer](https://github.com/NVIDIA/FasterTransformer/)

### Evaluation

* Settings
  * A real *12-hour availability trace* with *AWS g4dn spot instance* and extract *two representative 20-minute segments* with different dynamic behaviors
  * Two workloads
    * Stable inference request arrival workload
      * Different request arrival rates for different models
        * 1.5 requests/s for OPT-6.7B
        * 0.35 requests/s for GPT-20B
        * 0.2 requests/s for LLaMA-30B
      * *Gamma request arrival process* with a coefficient of variance of 6
    * Fluctuating inference request arrival workload
      * Trace: [Serverless-in-the-wild](https://www.usenix.org/conference/atc20/presentation/shahrad)
  * The maximum batch size $$B$$ is selected from $${1,2,4,8}$$
  * $$𝑆\_{𝑖𝑛}$$ is 512 — the sequence length of the input tokens
  * $$𝑆\_{𝑜𝑢𝑡}$$ is 128 — the sequence length of output tokens
* Baselines
  * **Rerouting** — dynamically reroutes interrupted requests to other available pipelines when preemption happens
  * **Reparallelization** — restart and reinitialize all instances without context migration
* Metrics
  * The average and various tail latencies
  * Monetary cost — USD/token

### Limitations and future work

* Strongly rely on the grace period → Can explore more solutions to improve system performance (e.g., inference workload prediction, instance availability prediction)
* Focus on single-type GPU instances → Can integrate heterogeneous spot instances or instances from different clouds
* Take inference latency minimization as the optimization target → Can explore other targets (e.g., strict SLO, high throughput)
* Can generalize to other preemptible resources (e.g., resource scheduler may preempt resources for urgent jobs with switching overheads)


# EuroSys 2024

## Meta Info

Homepage: <https://www.eurosys.org/news/eurosys-2024>

Acceptance rate: 14.7% (= 71 / 484)

## Papers

### GPU Sharing

* Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications \[[Personal Notes](/reading-notes/conference/eurosys-2024/orion-interference-aware-fine-grained-gpu-sharing-for-ml-applications)] \[[Paper](https://anakli.inf.ethz.ch/papers/orion_eurosys24.pdf)]
  * ETH

### Resource Scheduler for DL

* Blox: A Modular Toolkit for Deep Learning Schedulers \[[arXiv](https://arxiv.org/abs/2312.12621)] \[[Code](https://github.com/msr-fiddle/blox)]
  * UW-Madison & MSR

### Fault Tolerance for DL Training

* Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures \[[Personal Notes](/reading-notes/conference/eurosys-2024/jit-checkpointing)] \[[Paper](https://dl.acm.org/doi/10.1145/3627703.3650085)] \[[Homepage](https://www.microsoft.com/en-us/research/publication/just-in-time-checkpointing-low-cost-error-recovery-from-deep-learning-training-failures/)]
  * MSR India
  * Creates recovery state only when a failure occurs, avoiding periodic checkpoint-frequency tuning and steady-state checkpoint overhead.
  * Recovers by replaying a single minibatch iteration across GPUs, reducing failure recovery from minutes to seconds per GPU with nearly zero steady-state overhead.


# Orion: Interference-aware, fine-grained GPU sharing for ML applications

## Meta Info

Presented in [EuroSys 2024](https://anakli.inf.ethz.ch/papers/orion_eurosys24.pdf).

## Understanding the paper

* Orion — a system that transparently *intercepts GPU kernel launches* from multiple clients sharing a GPU
* It schedules work on the GPU *at the granularity of individual operators* and minimizes interference by taking into account *each operator’s compute and memory requirements*
* Integrated into PyTorch

### Technical details

* Influence the behavior of the hardware scheduler by using CUDA stream priorities
* CUDA Events to *monitor the progress of each stream* in the GPU
* Schedule each `cudaMemcpy` operation by *considering its PCIe bandwidth requirements and current bus bandwidth utilization*
* Use [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute) and [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) to collect *the compute throughput, memory throughput, and execution time of each kernel*

### Evaluation

* Baselines
  * Temporal sharing — time-slice the GPU by executing one job’s request at a time
  * NVIDIA MPS
  * CUDA Streams
  * [REEF](https://www.usenix.org/conference/osdi22/presentation/han)


# Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures

## Meta Info

Presented in [EuroSys 2024](https://doi.org/10.1145/3627703.3650085).

Authors: Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu (*MSR India*)

Homepage: <https://www.microsoft.com/en-us/research/publication/just-in-time-checkpointing-low-cost-error-recovery-from-deep-learning-training-failures/>

## Understanding the paper

### TL;DR

**Just-In-Time (JIT) Checkpointing** moves checkpoint creation from the normal training path to the failure path. Instead of periodically materializing large model and optimizer states, it constructs recovery state after a failure and rolls the distributed job back by only one minibatch iteration.

The key insight is to exploit the structure already present in synchronous training. A minibatch boundary is a natural consistency point: all ranks can replay the same iteration and resume from a globally consistent state. This changes recovery from “redo everything since the last periodic checkpoint” to “redo one iteration,” while avoiding checkpoint I/O when no failure occurs.

### Key design

* **Failure-triggered recovery state.** The system preserves enough information during execution to reconstruct a checkpoint only when a failure is detected. The expensive materialization step therefore leaves the steady-state training path.
* **Single-iteration rollback.** Recovery returns every worker to the beginning of the failed minibatch. All GPUs replay that iteration together, restoring the synchronization and parameter state expected by the next step.
* **Replay context.** Correct replay requires the minibatch identity and the relevant execution state, including rank placement and other nondeterministic inputs, to be reproducible after workers restart.
* **Transparent deployment.** The paper considers both training-code support and infrastructure-level mechanisms, aiming to provide JIT checkpointing without requiring users to redesign their training programs.

The resulting cost model differs from periodic checkpointing. Periodic schemes continuously trade checkpoint frequency against expected lost work: short intervals waste I/O, while long intervals increase replay after failures. JIT checkpointing pays primarily when a failure actually occurs—constructing recovery state, coordinating workers, and replaying one step—so it avoids choosing a checkpoint interval under changing failure rates.

### Evidence

The evaluation reports nearly zero steady-state checkpoint overhead. Recovery is reduced from several minutes of lost work to a few seconds per GPU because replay is bounded to one minibatch rather than the time since an earlier checkpoint.

The systems insight matters more than the absolute number: checkpoint cost should be evaluated across the whole GPU fleet. At large scale, repeatedly pausing every healthy GPU to protect against occasional failures can consume more resources than moving work into the uncommon recovery path.

### Caveats

* The approach assumes enough state survives a failure to construct a valid recovery point; correlated or cluster-wide failures may violate that assumption.
* Single-iteration replay must reproduce data-loader position, random state, rank mapping, and distributed execution consistently.
* JIT checkpoints do not fully replace durable periodic checkpoints needed for disaster recovery, long-term restart, or experiment reproducibility.
* Transparent deployment still requires integration with the training runtime and cluster control plane.


# NSDI 2024

## Meta Info

Homepage: <https://www.usenix.org/conference/nsdi24>

Paper list: <https://www.usenix.org/conference/nsdi24/technical-sessions>

## Papers

### Resource Management

* Resiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/zu)]
  * Google
  * Experience in designing and operating the software infrastructure that allows TPUv4 supercomputers to operate at scale.
* Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservices \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/wang-zibo)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-wang_zibo.pdf)] \[[Code](https://github.com/microsoft/autothrottle)]
  * USTC & ETH & MSR
  * Minimize CPU allocation of *microservice applications* while meeting SLO.
  * Service-level (low overhead & fast reaction) vs. Application-level (global visibility)
    * Captains (service-level): control based on throttle ratio target; collect data every 100ms, adjust allocation every 1s.
    * Tower (application-level): determine the best throttle targets for Captains to achieve; online learning (contextual bandit algorithm); one step per minute, each step runs in \~100ms.
* CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/rajasekaran)]
  * MIT & UT-Austin
  * Consider the communication pattern of different jobs while placing them on network links.

### Large Language Models (LLMs)

* LLM characterization
  * Characterization of Large Language Model Development in the Datacenter \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/hu)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-hu.pdf)] \[[Trace](https://github.com/InternLM/AcmeTrace)]
    * NTU & PKU & CUHK & Shanghai AI Lab
* LLM training
  * MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/jiang-ziheng)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-jiang_ziheng.pdf)] \[[Code](https://github.com/volcengine/veScale)]
    * ByteDance & PKU

### Utilize Spot Instances

* Can't Be Late: Optimizing Spot Instance Savings under Deadlines \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/wu-zhanghao)] \[[Trace](https://github.com/skypilot-org/spot-traces)]
  * UC Berkeley
  * **Outstanding Paper**
  * Characterization (e.g., availability, pricing, duration) of three-month-long spot availability traces on AWS.
  * **Uniform Progress**: a policy to make uniform progress towards the deadline, by distributing the job computation uniformly across the time.
* Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/duan)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-duan.pdf)] \[[Code](https://github.com/JF-D/Parcae)]
  * CUHK & ByteDance & CMU & UCLA & Microsoft
  * Proactively adjust the parallelization strategy of a DNN training job for future preemptions to maximize preemption-aware throughput (i.e., liveput).

### Multimodal Models

* DISTMM: Accelerating Distributed Multimodal Model Training \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/huang)]
  * Ohio State University & AWS
  * Partition and parallelize the submodules of a multimodal model based on their modalities and redistribute the training data.

### Diffusion Models

* Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/agarwal-shubham)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-agarwal_shubham.pdf)]
  * Adobe Research & UIUC
  * Approximate caching: reduce a certain number of denoising steps by reusing intermediate noise states created during a prior image generation.

### Deep Learning Recommendation Models (DLRMs)

* Accelerating Neural Recommendation Training with Embedding Scheduling \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/zeng)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-zeng.pdf)] \[[Code](https://github.com/HKUST-SING/herald)]
  * HKUST
  * **Herald**: an adaptive location-aware inputs allocator to determine *where embeddings should be trained* and an optimal communication plan generator to determine *which embeddings should be synchronized*.

### Fair Resource Allocation

* Solving Max-Min Fair Resource Allocations Quickly on Large Graphs \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/namyar-solving)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides_namyar-solving.pdf)] \[[Code](https://github.com/microsoft/Soroush)]
  * Microsoft & USC & Rice
  * **Soroush**: Single-Shot Max-Min Fair Allocator.
  * Deployed on Microsoft WAN.

### Network Emulation

* Crescent: Emulating Heterogeneous Production Network at Scale \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/gao-zhaoyu)] \[[Slides](https://www.usenix.org/system/files/nsdi24_slides-gao_zhaoyu.pdf)]
  * ByteDance & Cornell
  * **Crescent**: ByteDance’s *network emulation* platform for preventing *change-induced network incidents*.

### RDMA

* Harmonic: Hardware-assisted RDMA Performance Isolation for Public Clouds \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/lou)]
  * UIUC & Duke & Microsoft
  * **Harmonic**: microarchitecture-resource-aware RDMA performance isolation; including a programmable intelligent PCIe switch (prototyped with FPGA) and an RDMA-friendly rate limiter.

### PCIe

* Understanding Routable PCIe Performance for Composable Infrastructures \[[Paper](https://www.usenix.org/conference/nsdi24/presentation/hou)]
  * UW-Madison & ZJU
  * **rPCIeBench**: a software-hardware co-designed benchmarking framework to systematically characterize the *routable PCIe fabric*.


# NeurIPS 2023

## Meta Info

Homepage: <https://nips.cc/Conferences/2023>

Paper List: <https://nips.cc/virtual/2023/papers.html?filter=titles>


# SC 2023

## Meta Info

Homepage: <https://sc23.supercomputing.org/>

Paper list: <https://dl.acm.org/doi/proceedings/10.1145/3581784>

## Papers

### Distributed Training

* EasyScale: Elastic Training with Consistent Accuracy and Improved Utilization on GPUs \[[Paper](https://doi.org/10.1145/3581784.3607054)] \[[Code](https://github.com/sUntvoOk/EasyScale_info_for_SC23)]
  * BUAA & Alibaba
* Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency \[[Paper](https://doi.org/10.1145/3581784.3607073)] \[[Code](https://github.com/MaruyamaAya/Wpipe)]
  * NUS

### GPU Sharing

* Interference-aware Multiplexing for Deep Learning in GPU Clusters: A Middleware Approach \[[Personal Notes](/reading-notes/conference/sc-2023/iadeep)] \[[Paper](https://doi.org/10.1145/3581784.3607060)] \[[Code](https://github.com/buzy-coder/IADeep)]
  * UMacau & SIAT, CAS
  * IADeep — a cluster scheduler to co-locate DL training tasks

### Serverless Functions

* Rethinking Deployment for Serverless Functions: A Performance-first Perspective \[[Paper](https://doi.org/10.1145/3581784.3613211)] \[[Code](https://github.com/tjulym/Chiron)]
  * TJU
  * Chiron


# Interference-aware multiplexing for deep learning in GPU clusters: A middleware approach

## Meta Info

Presented in [SC 2023](https://doi.org/10.1145/3581784.3607060).

## Understanding the paper

### Opportunities in co-locating DL training tasks

* Tune training configurations (e.g., batch size) across all co-located tasks
* Choose appropriate tasks to multiplex on a GPU device

### Challenges

* Trade-off between *mitigating interference* and *accelerating training progress* to achieve optimal training time
* Vast search space of task configurations
* Coupling between *adjusting task configurations* and *designing task placement policies*


# SoCC 2023

## Meta Info

Homepage: <https://acmsocc.org/2023/>

Paper list: <https://acmsocc.org/2023/accepted-papers.html>

## Papers

### Resource Allocation

* Lifting the Fog of Uncertainties: Dynamic Resource Orchestration for the Containerized Cloud \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624646)]
  * UofT
  * Adaptively configure resource parameters
  * Built on contextual bandit techniques
  * Balance between performance and resource cost
* Not All Resources are Visible: Exploiting Fragmented Shadow Resources in Shared-State Scheduler Architecture \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624650)]
  * SJTU & Huawei
  * Shared-state schedulers: A central state view *periodically* updates the *global* cluster status to *distributed* schedulers
  * Shadow resources: Resources invisible to *shared-state schedulers* until the next view update
  * Resource Miner (RMiner) includes a *shadow resource manager* to manage shadow resources, an *RM filter* to select suitable tasks as RM tasks, an *RM scheduler* to allocate shadow resources to RM tasks
* Gödel: Unified large-scale resource management and scheduling at ByteDance \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624663)]
  * ByteDance & UVA
  * Industry Paper
  * A unified infrastructure for all business groups to run their diverse workloads
  * Built upon Kubernetes

### Machine Learning

* Anticipatory Resource Allocation for ML Training Clusters \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624669)]
  * Microsoft Research & UW
  * Schedule based on *predictions of future job arrivals and durations*
  * Deal with prediction errors
* tf.data service: A Case for Disaggregating ML Input Data Processing \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624666)]
  * Google & ETH
  * Industry Paper
  * A disaggregated input data processing service built on top of tf.data in TensorFlow
  * Horizontally scale out to right-size host resources (CPU/RAM) for data processing in each job
  * Share ephemeral preprocessed data results across jobs
  * Coordinated reads to avoid stragglers
* Is Machine Learning Necessary for Cloud Resource Usage Forecasting? \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624790)]
  * IMDEA Software Institute
  * Vision Paper
  * Question: Whether *complex machine learning models* are necessary to use?
  * Proposal: Practical memory management systems need to first identify the extent to which simple solutions can be effective.

### Serverless Computing

* Golgi: Performance-Aware, Resource-Efficient Function Scheduling for Serverless Computing \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624645)]
  * HKUST & WeBank
  * **Best Paper Award!**
  * A scheduling system for serverless functions to *minimize resource provisioning costs* while *meeting the function latency requirements*
  * Overcommit functions based on their past resource usage; Identify nine low-level metrics (e.g., request load, resource allocation, contention on shared resources); Use the Mondrian Forest to predict the function performance
  * Employ a conservative exploration-exploitation strategy for request routing; By default, route requests to non-overcommitted instances; Explore to use overcommitted instances
  * Vertical scaling to dynamically adjust the concurrency of overcommitted instances
* Parrotfish: Parametric Regression for Optimizing Serverless Functions \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624654)]
  * UBC & UTokyo & INSAT
  * Find optimal configurations through an online learning process
  * Use parametric regression to choose the right memory configurations for serverless functions
* AsyFunc: A High-Performance and Resource-Efficient Serverless Inference System via Asymmetric Functions \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624664)] \[[Code](https://github.com/peiqiangyu/AsyFunc)]
  * HUST & Huawei & Peng Cheng Laboratory
  * Problem: The time-consuming and resource-hungry model-loading process when scaling out function instances
  * Observation: The sensitivity of each layer to the computing resources is mostly anti-correlated with its memory resource usage
  * Asymmetric Functions
    * The original Body Function loads a complete model to meet stable demands
    * The proposed lightweight Shadow Function only loads a portion of resource-sensitive layers to deal with sudden demands effortlessly
  * AsyFunc — an inference serving system with an auto-scaling and scheduling engine; Built on top of Knative
* Chitu: Accelerating Serverless Workflows with Asynchronous State Replication Pipeline \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624794)] \[[Code](https://github.com/sigserverless/chitu)]
  * ISCAS & ICT, CAS
  * **Asynchronous State Replication Pipelines (ASRP)** to speed up serverless workflows for general applications
  * Three insights
    * Provide differentiable data types (DDT) at the programming model level to support incremental state sharing and computation
    * Continuously deliver changes of DDT objects in real-time
    * Direct communication and change propagation
  * Built atop OpenFaaS
* How Does It Function? Characterizing Long-term Trends in Production Serverless Workloads \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624783)] \[[Trace](https://github.com/sir-lab/data-release)]
  * Huawei
  * Industry Paper
  * Two new serverless traces in Huawei Cloud
    * The first trace: Huawei's *internal* workloads; Per-second statistics for 200 functions
    * The second trace: Huawei's public FaaS platform; Per-minute arrival rates for over 5000 functions
  * Characterize resource consumption, cold-start times, programming languages used, periodicity, per-second versus per-minute burstiness, correlations, and popularity.
  * Findings
    * Requests vary by up to 9 orders of magnitude across functions, with some functions executed over 1 billion times per day
    * Scheduling time, execution time and cold-start distributions vary across 2 to 4 orders of magnitude and have very long tails
    * Function invocation counts demonstrate strong periodicity for many individual functions and on an aggregate level
  * The need for further research in *estimating resource reservations and time-series prediction*
* Function as a Function \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624648)]
  * ETH
  * Vision Paper
  * Dandelion -- a clean state FaaS system; Treat serverless functions as pure functions; Explicitly separate computation and I/O; Hardware acceleration; Enable dataflow-aware function orchestration
* The Gap Between Serverless Research and Real-world Systems \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624785)]
  * SJTU & Huawei Cloud
  * Vision Paper
  * Five open challenges
    * Optimize cold start latency: Most existing works only consider synchronous starts; Asynchronous start in Industry
    * Declarative approach: Whether Kubernetes is the right system for serverless computing?
    * Scheduling cost
    * Balance different scheduling policies within a serverless system
    * Costs of sidecar

### Sustainable Computing

* Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale \[[Paper](https://dl.acm.org/doi/10.1145/3620678.3624793)]
  * MIT & NEU
  * Significant decreases in both temperature and power draw, reducing power consumption and potentially *improving hardware life-span*, with *minimal impact on job performance*


# SOSP 2023

## Meta Info

Homepage: <https://sosp2023.mpi-sws.org/>

## Papers

### Large Language Models (LLMs)

* Efficient Memory Management for Large Language Model Serving with PagedAttention \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613165)] \[[arXiv](https://browse.arxiv.org/abs/2309.06180)] \[[Code](https://github.com/vllm-project/vllm)] \[[Homepage](https://vllm.ai/)]
  * UC Berkeley & Stanford & UCSD
  * vLLM, PagedAttention
* Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates \[[Paper](https://dl.acm.org/doi/abs/10.1145/3600006.3613152)] \[[arXiv](https://browse.arxiv.org/abs/2309.08125)] \[[Code](https://github.com/SymbioticLab/Oobleck)]
  * UMich SymbioticLab & AWS & PKU
* Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613145)]
  * Rice & AWS

### Deep Learning Recommendation Models (DLRMs)

* UGache: A Unified GPU Cache for Embedding-based Deep Learning \[[Personal Notes](/reading-notes/conference/sosp-2023/ugache)] \[[Paper](https://dl.acm.org/doi/10.1145/3600006.3613169)]
  * SJTU
  * Multi-GPU embedding cache; exploit cross-GPU interconnects (NVLink, NVSwitch).
* Bagpipe: Accelerating Deep Recommendation Model Training \[[Paper](https://dl.acm.org/doi/abs/10.1145/3600006.3613142)]
  * UW-Madison & UChicago


# UGache: A unified GPU cache for embedding-based deep learning

\#DLRM\_inference #GPU\_embedding\_cache

## Meta Info

Presented in [SOSP 2023](https://doi.org/10.1145/3600006.3613169).

Authors: Xiaoniu Song, Yiwen Zhang, Rong Chen, Haibo Chen (*SJTU*)

## Understanding the paper

### TL;DR

* UGache — A unified cache across multiple GPUs
* Exploit *cross-GPU interconnects* (e.g., NVLink, NVSwitch)
  * Provide an optimal data placement with a unified abstraction for various GPU interconnects and bandwidths

### Background

* Performance bottleneck
  * Fetch embedding entries
    * Example: GNN training (31.3 ms)
      * Embedding extraction: 20.7 ms
      * Fetch missing data from host memory: 17.9 ms (out of 20.7 ms)
  * Memory limitation in a single GPU

### Limitation of prior works

* Single GPU embedding cache
  * Only let each GPU cache the hottest embeddings independently
  * The hit rate is limited due to the single GPU’s memory capacity, and the time of embedding extraction still dominates the entire model
* Multi-GPU embedding cache
  * *Fail* to address challenges related to *cache policy* and *extraction mechanism*
  * *Replication* cache
    * Directly port single-GPU cache to multi-GPU platform → Each GPU caches the hottest entries independently
    * Drawbacks
      * Each GPU covers similar requests
      * Waste bandwidth across GPUs
  * *Partition* cache
    * Cache as many individual entries as possible and serve the majority of accesses through *fast GPU interconnects*
    * Drawbacks
      * Increasing cache capacities using a partition policy does not lead to proportional increases in hit rates

### Characteristics of Embedding Access

* Read-only
  * Inference
  * Pre-training → The embedding table is trained beforehand and distributed across different downstream workloads
* Batched access
  * Only access a subset of embedding entries using batched sparse inputs as keys
* Skewed hotness
  * Power-law distribution
* Predictable pattern workload
  * The skewness of accessing embeddings is predictable and stable

### System design

* **Extractor**
  * A factored extraction mechanism to extract embedding entries from *multiple sources*
  * Statically dedicate GPU cores to access different sources
* **Solver**
  * Find a balance between *caching more distinct entries to improve global hit rate* and *caching more replicas to improve local hit rate*
  * Define *a hotness metric* to measure the access frequency for each entry
  * *Profile* hardware platform’s information to *estimate embedding extraction time*
  * Utilize Mixed-Integer Linear Programming to solve a cache policy to minimize the extraction time

### Implementation

* Integrated into TensorFlow and PyTorch.

### Evaluation

* Two representative applications: DLRM inference, GNN training


# SIGCOMM 2023

## Meta Info

Homepage: <https://conferences.sigcomm.org/sigcomm/2023/>

## Papers

### Mixture of Experts (MoE)

* Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models \[[Paper](https://dl.acm.org/doi/10.1145/3603269.3604869)]
  * THU & ByteDance


# HotChips 2023

## Meta Info

Homepage: <https://hc2023.hotchips.org/>


# ICML 2023

## Meta Info

Homepage: <https://icml.cc/Conferences/2022>

Paper List: <https://icml.cc/virtual/2023/papers.html?filter=titles>

## Papers

### LLM Inference

* Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time \[[Paper](https://proceedings.mlr.press/v202/liu23am.html)]
* FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU \[[Personal Notes](/reading-notes/miscellaneous/arxiv/2023/flexgen)] \[[Paper](https://proceedings.mlr.press/v202/sheng23a.html)]


# ATC 2023

## Meta Info

2023 USENIX Annual Technical Conference

Homepage: <https://www.usenix.org/conference/atc23>

Paper List: <https://www.usenix.org/conference/atc23/technical-sessions>

## Papers

### Mixture of Experts (MoE)

* [Accelerating Distributed MoE Training and Inference with Lina](https://www.usenix.org/conference/atc23/presentation/li-jiamin)
* [SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization](https://www.usenix.org/conference/atc23/presentation/zhai)

### GPU Fragmentation

* [Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent](https://www.usenix.org/conference/atc23/presentation/weng)


# Accelerating Distributed MoE Training and Inference with Lina




---

[Next Page](/llms-full.txt/1)