Large Language Model (LLM)
I am actively maintaining this list.
LLM Training
Hybrid Parallelism
RL Post-Training
Reading map. These works differ along six axes:
Disaggregated RL systems: split rollout, inference, environment, reward, and training stages across best-fit resources.
Workflow scheduling and resource reallocation: reshape RL pipelines or dynamically move compute, memory, communication, and parameters across roles.
Rollout latency and long-tail mitigation: predict, batch, reuse, or otherwise reduce long-tail rollout work.
Speculative decoding for RL: adapt draft-and-verify generation to RL training constraints such as drafter staleness and rollout distribution shift.
Heterogeneous environments: make RL training efficient across mixed GPU generations and hardware capabilities.
Fault tolerance: isolate and recover failures across trainer, rollout, and control-plane roles.
RLinf: Flexible and Efficient Large-Scale Reinforcement Learning via Macro-to-Micro Flow Transformation (OSDI 2026) [Paper]
THU & Infinigence-AI & PKU & UC Berkeley & Zhongguancun Academy & BUAA & SJTU
Axis: Workflow scheduling and resource reallocation
Transforms RL workflows from macro pipelines into micro flows to improve scheduling flexibility and hardware utilization.
DynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training (OSDI 2026) [Paper]
PKU & Infinigence-AI & ICT, CAS & BUAA & THU & SJTU
Axis: Workflow scheduling and resource reallocation
Dynamically reallocates compute, memory, and communication resources across heterogeneous RL components.
Beat the long tail: Distribution-Aware Speculative Decoding for RL Training (MLSys 2026) [Paper]
Together AI & Stanford & UCSD
Axis: Speculative decoding for RL; Rollout latency and long-tail mitigation
Identifies rollout length long-tail as a bottleneck and applies distribution-aware speculative decoding to accelerate generation.
HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments (MLSys 2026) [Paper]
Amazon
Axis: Heterogeneous environments
Optimizes LLM reinforcement learning across heterogeneous GPU environments with multi-generation hardware.
ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems (MLSys 2026) [Paper]
NTU & HPC-AI Tech
Axis: Speculative decoding for RL
Addresses drafter staleness and draft-verification misalignment when integrating speculative decoding into RL systems.
Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter (ASPLOS 2026) [Paper] [arXiv] [Code]
MIT & NVIDIA & ETH Zurich & MIT-IBM AI Lab & UMass Amherst
Axis: Speculative decoding for RL; Rollout latency and long-tail mitigation
Uses TLT to accelerate reasoning RL rollouts with adaptive speculative decoding, training a lightweight drafter on idle GPUs and selecting rollout strategies per batch.
History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL (ASPLOS 2026) [Paper]
SJTU & ByteDance
Axis: Rollout latency and long-tail mitigation
Exploits reusable structure across rollout histories to reduce redundant rollout work.
ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation (MLSys 2025) [Paper] [arXiv] [Code]
THU
Axis: Workflow scheduling and resource reallocation
Reliability and Fault Tolerance
Reading map. These works differ along four axes:
Recovery mechanisms: reduce lost work after failures through redundancy, checkpoint placement, or pre-planned reconfiguration.
Production reliability infrastructure: make failures observable, diagnosable, and routinely recoverable at 10K+ GPU scale.
Empirical reliability studies: characterize production failure modes and operational mitigations from large training runs.
Workload resilience: absorb dynamic workload variation before it turns into large efficiency loss or training instability.
SDCs in the Wild: Characterizing and Diagnosing SDC-Defective GPUs in Production LLM Training (OSDI 2026) [Personal Notes] [Paper]
SJTU & ByteDance Seed
Axis: Empirical reliability studies; Production reliability infrastructure
Characterizes 23 SDC-defective GPUs from production LLM training and shows synthetic microbenchmarks miss over 60% of defective devices because SDCs can be aging-related, data-dependent, unit-specific, and invisible to ECC/thermal protections.
Introduces SDCHunter, replaying the exact training workload and input data that triggered the failure to diagnose defective GPUs.
Deployed at ByteDance to mitigate 40 SDC incidents, with less than 4% training overhead, 100% coverage and device-level localization for real SDC-defective accelerators, and diagnosis shortened to within one hour.
Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours (OSDI 2026) [Personal Notes] [Paper]
THU & ByteDance
Axis: Empirical reliability studies; Production reliability infrastructure
Presents AEGIS, an online SDC detection framework that decouples lightweight corruption sensing from definitive verification during large-scale LLM training.
Uses a two-stage
cSensor-cVerifierdesign to keep runtime monitoring lightweight while confirming suspected silent corruptions.Deployed over 35 million GPU hours in production, detecting 18 SDC incidents and 13 faulty GPUs with 0.86% overhead.
MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production (EuroSys 2026) [Paper] [arXiv]
SJTU & ByteDance
Axis: Workload resilience
MLLM training can lose efficiency even without hard failures when dynamic modality and length mixtures break static encoder-backbone parallelism; decouple encoder/LLM parallelism and rebalance data loading and resharding.
Robust LLM Training Infrastructure at ByteDance (SOSP 2025) [Paper] [arXiv]
HKU & ByteDance Seed
Axis: Production reliability infrastructure
Large-scale LLM training needs training-aware fault management, not only generic cluster monitoring; exploit parallelism structure and training characteristics for high-capacity tolerance, fault demarcation, and localization.
Large-Scale AI Infra Reliability: Challenges, Strategies, and Llama 3 Training Experience (DSN-S 2025) [Paper]
Meta
Axis: Empirical reliability studies
16K-GPU training reliability is dominated by hardware failure attribution and mitigation; the contribution is a failure taxonomy and operational strategies rather than a new recovery mechanism.
Holmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters (NSDI 2025) [Paper]
FDU & Tencent & UChicago
Axis: Production reliability infrastructure
Silent irregular iterations can cost more training time than fail-stop failures; Holmes localizes them in real time using abnormal communication-operator detection, communication-operator graphs, and cross-iteration analysis.
Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures (EuroSys 2024) [Personal Notes] [Paper] [Homepage]
MSR India
Axis: Recovery mechanisms
Creates recovery state only when a failure occurs, avoiding periodic checkpoint-frequency tuning and steady-state checkpoint overhead.
Recovers by replaying a single minibatch iteration across GPUs, reducing failure recovery from minutes to seconds per GPU with nearly zero steady-state overhead.
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs (NSDI 2024) [Paper] [Slides] [Code]
ByteDance & PKU
Axis: Production reliability infrastructure
Stability at 10K+ GPU scale is a full-stack property; deep observability across system components is needed to identify root causes and turn them into fault-tolerance and straggler-mitigation techniques.
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates (SOSP 2023) [Paper] [arXiv] [Code]
UMich SymbioticLab & AWS & PKU
Axis: Recovery mechanisms
Pre-generate heterogeneous pipeline templates and instantiate
f + 1logically equivalent pipeline replicas, so failures can be handled by reconfiguration using already-replicated model states.
Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints (SOSP 2023) [Paper]
Rice & AWS
Axis: Recovery mechanisms
Remote storage bandwidth makes checkpoint-based recovery slow; place checkpoints in aggregate host CPU memory and schedule checkpoint traffic to preserve training throughput.
Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs (NSDI 2023) [Paper] [Code]
UCLA & CMU & MSR & Princeton
Axis: Recovery mechanisms
Pipeline bubbles can hide redundant neighboring-layer computation, giving preemptible-instance training low-pause recovery without paying the full cost of frequent checkpointing.
LLM Inference
Heterogeneous Deployment
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs (arXiv:2605.04357) [arXiv]
CMU & PKU
Jointly optimizes multi-model resource allocation and per-replica serving strategies across heterogeneous cloud GPUs under throughput demand, latency SLOs, price, and resource availability.
Uses a lossless two-stage decomposition to preserve joint optimality while making online re-solving practical as demand and availability change.
SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling (SIGMETRICS Abstracts 2026) [Paper] [arXiv] [Code]
UIUC & GaTech & IISc & Microsoft
Serves mixed latency-sensitive and latency-insensitive LLM workloads across cloud regions with forecast-aware routing, GPU VM scaling, and model placement.
Co-optimizes short-term request routing and longer-lead-time resource allocation with traffic forecasts and ILP scheduling.
Cauchy: A Cost-Efficient LLM Serving System through Adaptive Heterogeneous Deployment (SoCC 2025) [Paper]
BUAA & Kuaishou
Deploys prefill and decode computation onto suitable heterogeneous GPU combos according to workload phase demands and cost efficiency.
Combines combo selection, hierarchical request scheduling, and dynamic autoscaling to maintain SLOs under changing request rates.
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (arXiv:2502.00722) [arXiv]
Cambridge & HKUST & PKU & ETH & Purdue
Characterizes how workload mix, GPU composition, deployment configuration, and request assignment affect LLM serving cost-efficiency on heterogeneous cloud GPUs.
Uses mixed-integer linear programming to choose cost-efficient serving plans under price-budget and real-time GPU-availability constraints.
HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment (ICLR 2025) [Paper] [arXiv]
HKUST
Extends heterogeneous LLM serving to prefill-decode disaggregation, jointly placing phase computation and KV-cache communication across heterogeneous GPUs and links.
Combines graph partitioning and max-flow optimization to co-optimize resource allocation, per-phase parallel strategies, and inter-phase KV transfer.
HexGen: Generative Inference of Foundation Model over Heterogeneous Decentralized Environment (ICML 2024) [Personal Notes] [arXiv] [Code]
HKUST & ETH & CMU
Supports asymmetric tensor model parallelism and pipeline parallelism, allowing each pipeline stage to use a different layer count and tensor-parallel degree.
Formulates heterogeneous decentralized inference placement as a constrained optimization problem and searches layouts with a heuristic evolutionary algorithm.
SpotServe: Serving Generative Large Language Models on Preemptible Instances (ASPLOS 2024) [Personal Notes] [arXiv] [Code]
CMU & PKU & CUHK
Dynamically adapts LLM parallelization configurations under changing preemptible-instance availability and workload fluctuation.
Uses migration planning and stateful inference recovery to reduce tail latency and monetary cost on volatile GPU capacity.
Parallelism and Partitioning
Efficiently Scaling Transformer Inference (MLSys 2023) [Paper]
Google
Outstanding Paper Award
Model partitioning; PaLM; TPUv4
LLM-based Applications
Teola: Towards End-to-End Optimization of LLM-based Applications (ASPLOS 2025) [arXiv]
CUHK
An orchestration framework for LLM-based applications: utilize task primitives as the basic units; represent each query’s workflow as a primitive-level dataflow graph.
Enable larger design space for optimization including graph optimization (i.e., parallelization and pipelining) and application-aware scheduling.
SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) [Personal Notes] [Paper] [arXiv] [Code]
UC Berkeley & Stanford
Co-design the front-end programming interface and back-end serving runtime
SGLang; SGVM w/ RadixAttention
Reuse KV cache across multiple calls and programs
Agentic Inference
ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System (ICML 2026) [Paper] [arXiv] [Code] [Blog]
GaTech & UIUC & CMU & Together AI
Spotlight
Introduce ThunderAgent, an agentic inference and rollout system that represents workflows as LLM Programs with unified KV-cache, state, and tool-resource metadata.
Use program-aware scheduling and tool resource management to improve KV-cache reuse, reduce memory imbalance, and prepare tool environments asynchronously.
Workload Characterization
TraceLab: Characterizing Coding Agent Workloads for LLM Serving (arXiv:2606.30560) [arXiv] [Code] [Homepage] [Blog]
UW & Wuhan University of Technology & SJTU
Release a coding-agent trace with 4,265 sessions, 357,161 agent steps, and 432,510 tool calls from Claude Code and Codex usage.
Characterize long autonomous loops, long contexts with short outputs, heavy-tailed tool calls, and high but imperfect prefix-cache hit rates.
Identify serving opportunities around lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and KV-cache management around human-paced gaps.
Measuring Agents in Production (ICML 2026) [Paper] [arXiv]
UC Berkeley & Intesa Sanpaolo & UIUC & Stanford & IBM Research
Oral
Present MAP, a systematic study of production LLM agents using 20 interviews and a survey of 86 deployed-systems practitioners across 26 domains.
Find that production agents often use simple, controllable approaches: bounded step counts before human intervention, off-the-shelf prompting, and human evaluation.
Retrieval-Augmented Generation (RAG)
CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation (arXiv:2502.11101) [arXiv]
Jeonbuk National University & Seoul National University
Leverage query-independent, offline caching to reuse a context KV cache store.
Cache Re-Positioning: shift keys to different positions in the encoding space.
Layer-Adaptive Cache Pruning: discard low-relevance caches for documents during pre-filling.
Adaptive Positional Allocation: adjust cache positions to maximize the use of the available positional encoding range.
Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation (SIGMOD 2025) [arXiv]
Adobe Research & IIT Bombay & IIT Kanpur
Identify the reusability of chunk-caches; perform a small fraction of recomputation to fix the cache to maintain output quality; store and evict chunk-caches.
A wrapper around vLLM; built on Xformers backend optimized with Triton.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation (arXiv:2404.12457) [arXiv]
PKU & ByteDance
Organize the intermediate states of retrieved knowledge in a knowledge tree; cache them in the GPU and host memory.
Replacement policy: evaluate each node based on its access frequency, size, and access cost.
Priority= Clock + (Frequency × Cost Size) / Size
Nodes with lower priority are evicted first.
Built on vLLM.
Request Scheduling
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference (NSDI 2026) [Paper] [arXiv] [Code]
PKU
Enables iteration-level preemptive scheduling for autoregressive decoding instead of request-level FIFO execution.
Combines a skip-join multi-level feedback queue scheduler with proactive intermediate-state swapping to reduce head-of-line blocking.
Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) [Personal Notes] [Paper]
Seoul National University & FriendliAI
Iteration-level scheduling; selective batching.
KV Cache Management
DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants (NSDI 2026) [Paper] [arXiv]
UChicago & Microsoft
Reuses prefix KV caches across fine-tuned LLM variants with the same architecture, including across distributed nodes.
Selectively recomputes a small subset of layers from another model's KV cache and reuses the remaining layers with negligible quality loss.
Jenga: Effective Memory Management for Serving LLM with Heterogeneity (SOSP 2025) [Personal Notes] [Paper] [arXiv]
THU & UChicago & UC Berkeley
Heterogeneous embedding sizes, attention mechanisms, and token-dependency patterns in modern LLMs break fixed-page KV-cache assumptions and create fragmentation.
Uses a two-level memory allocator with LCM-sized compatible pages and layer-specific caching/eviction policies for heterogeneous attention patterns.
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion (EuroSys 2025) [Paper] [arXiv] [Code]
CUHK-Shenzhen & UChicago & Stanford
Best Paper Award (Spring)
Combines multiple precomputed KV caches for RAG inputs while selectively recomputing a small fraction of tokens to recover cross-attention.
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving (SIGCOMM 2024) [arXiv] [Code] [Video]
UChicago & Microsoft & Stanford
Prompt Cache: Modular Attention Reuse for Low-Latency Inference (MLSys 2024) [Paper] [arXiv]
Yale & Google
Precomputes and stores attention states for reusable prompt modules such as system messages, templates, and context documents.
Uses a schema to preserve positional correctness during attention-state reuse and reduce time-to-first-token without model changes.
ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching (ISCA 2024)
Prefill-Decode (PD) Disaggregation
Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter (arXiv:2604.15039) [arXiv]
Moonshot AI & THU
Introduce PrfaaS, a cross-datacenter serving architecture that selectively offloads long-context prefills to standalone compute-dense clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode.
Combine model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement instead of fully externalizing all prefill requests.
Remove the requirement that heterogeneous accelerators share a single low-latency RDMA fabric; on an internal 1T-parameter hybrid model, improve serving throughput by 54% over homogeneous PD and 32% over a naive heterogeneous baseline.
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST 2025) [Paper] [arXiv] [Slides] [Code]
Mootshot AI & Tsinghua
Best Paper Award
Separate the prefill and decoding clusters; prediction-based early rejection.
Distributed multi-layer KVCache pool; prefix-hashed KVCache object storage.
Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (arXiv:2401.11181) [arXiv]
ICT, CAS & Huawei Cloud
Chunked Prefill
LMPrefill: An Inference Engine for Prefill-only Workloads in Large Language Model Applications (SOSP 2025) [Paper] [arXiv]
UChicago & THU & LinkedIn & UC Berkeley
Targets LLM applications that generate only one output token and avoids storing KV caches for all layers.
Combines hybrid prefilling, suffix KV cache discarding/offloading, and JCT-aware scheduling for prefill-only requests.
Serverless Inference
HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds (NSDI 2026) [Paper] [arXiv] [Code]
PKU & Alibaba Cloud
Minimizes serverless LLM cold-start latency in public clouds through proactive model distribution, overlapped worker startup, and GPU-network-contention-aware worker placement.
Consolidates pipelines to reduce cold-start resource usage while improving SLO attainment.
FaaScale: Unlocking Fast LLM Scaling for Serverless Inference (MLSys 2026) [Paper] [arXiv]
CUHK-SZ & UVA & HKUST & Alibaba & Nokia Bell Labs
Formerly λScale; enables fast model scaling for serverless LLM inference with pipelined multicast inference.
Uses PipeCast to adaptively multicast model blocks and dynamically form cross-node inference pipelines during model transfer.
LoRA Serving
CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference (arXiv:2401.11240) [arXiv]
HKUST & CUHK-Shenzhen & Shanghai AI Lab & Huawei Cloud
S-LoRA: Serving Thousands of Concurrent LoRA Adapters (MLSys 2024) [arXiv] [Code]
UC Berkeley
Punica: Multi-Tenant LoRA Serving (MLSys 2024) [arXiv] [Code]
UW & Duke
Position-Independent Caching (PIC)
EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models (ICML 2025) [arXiv]
PKU & NJU & Huawei Cloud
Key insight: the initial tokens of each chunk separately absorb a disproportionate amount of attention, preventing subsequent tokens from attending to relevant parts.
Propose an algorithm named LegoLink to recompute k (≤ 32) initial tokens on each chunk (except the first chunk) → Recognize their non-initial status and cripple their attention-absorbing ability.
Compared to CacheBlend, LegoLink reduces recomputation complexity and relies on static attention sparsity.
Compression
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization (ISCA 2024)
Sparsity
Speculative Decoding
SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification (ASPLOS 2024) [arXiv] [Code]
CMU
Speculative Decoding with Big Little Decoder (NeurIPS 2023) [Paper]
UC Berkeley & ICSI & LBNL
Offloading
LLM in a flash: Efficient Large Language Model Inference with Limited Memory (arXiv 2312.11514) [arXiv]
Apple
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU (ICML 2023) [Personal Notes] [Paper] [Code]
Stanford & UC Berkeley & ETH & Yandex & HSE & Meta & CMU
High-throughput serving; only use a single GPU.
Fairness
Locality-aware Fair Scheduling in LLM Serving (arXiv:2501.14312) [arXiv]
UC Berkeley
LLM Alignment
Acronyms
LLM: Large Language Model
LoRA: Low-Rank Adaptation
RL: Reinforcement Learning
RLHF: Reinforcement Learning from Human Feedback
SDC: Silent Data Corruption
Last updated