Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

NSDI 2026

Meta Info

Homepage: https://www.usenix.org/conference/nsdi26

Paper list: https://www.usenix.org/conference/nsdi26/technical-sessions

Acceptance Rate

  • Spring: 24.2% (= 50 / 207)

  • Fall: 22.1% (= 100 / 452)

Papers

Large Language Models (LLMs)

  • LLM Training

    • MoE Training

      • SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling [Paper] [arXiv]

        • Stanford & NVIDIA & OpenAI

        • Propose SYMI, an adaptive MoE training system that decouples the placement of expert parameters from their large optimizer states.

        • Statically partition optimizer states across training nodes while dynamically adjusting expert parameter placement using existing weight updates, avoiding frequent state migration overheads.

        • Improve time-to-convergence by 30.5% over DeepSpeed and 25.9% over FlexMoE.

      • Checkpoint Lite, Recover Right: Efficient Fault Tolerant Training of Mixture-of-Experts Models Using Sparse Checkpoints [Paper]

        • Stanford & NVIDIA

    • Cross-Cluster Training

      • Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-Cluster LLM Training at Scale [Paper]

        • NUDT & Shanghai AI Lab & NTU

    • Fault Tolerance

      • Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training [Paper] [arXiv]

        • HKUST & Alibaba

        • Present a straggler-resilient hybrid-parallel training system for pipeline-parallel large-model training under communication stragglers.

        • Adapt the pipeline schedule with an analytical model to absorb slow communication without cascading bubbles, and offload delayed communication to host memory with CPU-side RDMA to avoid GPU head-of-line blocking.

        • Reduce training iteration time by 1.2x to 3.5x under various straggler settings.

      • Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale [Paper] [arXiv]

        • SJTU & Ant Group & NUS

        • Introduce Flare, a diagnostic framework for distributed LLM training at scale.

        • Combine a lightweight tracing daemon for full-stack, backend-extensible tracing with a diagnostic engine that automatically diagnoses anomalies, especially performance regressions.

        • Demonstrate deployment across 6,000 GPUs with continuous operation for over eight months in production scenarios.

      • EROICA: Online Performance Troubleshooting for Large-scale Model Training in Production [Paper] [arXiv]

        • Alibaba

        • Present EROICA, an online troubleshooting system for large-scale model training that combines fine-grained profiling with full-cluster coverage.

        • Summarize runtime execution patterns through online profiling and use differential observability to localize hardware, software, and mixed root causes with minimal production impact.

        • Report deployment on production GPU clusters of about 100,000 GPUs for 1.5 years, diagnosing difficult performance issues with 97.5% success.

    • RL Post-Training

      • RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching [Paper] [arXiv]

        • HKUST & Alibaba

        • Introduce tail batching, a rollout scheduling strategy for synchronous RL post-training that packs prompts with long-tail responses into a small number of long rounds while keeping most rounds balanced and short.

        • Combine tail batching with elastic rollout parallelism, dynamic reward-stage resource scheduling, and stream-based training to reduce rollout bubbles without relaxing synchronization.

        • Cut end-to-end training time by 2.03x to 2.56x over veRL and by up to 2.24x over RLHFuse on Qwen2.5 models.

    • Performance Modeling and Simulation

      • Supercharging Packet-Level Network Simulation of Large Model Training via Memoization and Fast-Forwarding [Paper] [arXiv]

        • THU & Zhongguancun Laboratory & BUPT & Huawei

        • Propose Wormhole, a user-transparent packet-level discrete-event simulation kernel for large-model training that reduces redundant simulation work without simplifying the network model.

        • Memoize unsteady states and fast-forward steady states through network partitioning, state reuse, and rate-based steady-state identification while preserving simulation consistency.

        • Achieve 744x speedup over ns-3 with less than 1% error; combined with multithreading, the speedup reaches 1012x.

      • GPUSynth: Maximizing Code Reuse in Simulation-Based Machine Learning System Performance Estimation [Paper] [arXiv]

        • Duke & UC Berkeley & Meta

        • Introduce Phantora, a hybrid GPU-cluster simulator for ML training performance estimation that runs unmodified ML frameworks in a distributed containerized environment.

        • Intercept GPU and communication operations during execution, enabling direct reuse of ML framework source code instead of reimplementing frameworks inside a simulator.

        • Match the accuracy of static workload simulation while supporting three state-of-the-art LLM training frameworks out of the box on a single GPU.

  • LLM Inference

    • Request Scheduling

      • FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference [Paper] [arXiv] [Code]

        • PKU

        • Propose FastServe, an LLM serving system that enables iteration-level preemptive scheduling for autoregressive decoding instead of request-level FIFO execution.

        • Combine a skip-join multi-level feedback queue scheduler with a parallelism-aware suspension strategy that downscales preempted models into CPU/NVMe storage.

        • Improve multi-model serving throughput and latency; the abstract reports up to 5.1x higher throughput and 13.9x to 111.8x lower latency than prior systems.

      • JITServe: SLO-aware LLM Serving with Imprecise Request Information [Paper] [arXiv]

        • UIUC & NJU & Google Labs & Cisco Research

        • Propose JITServe, an LLM serving system for workloads where request information is imprecise at arrival time.

        • Forecast future decoding lengths from near-future token generation and construct just-in-time execution schedules that balance prediction accuracy against rapidly changing system dynamics.

        • Increase throughput while improving latency SLO satisfaction; the abstract reports 1.8x to 7.5x higher throughput and 2.2x to 8.7x better SLO attainment than prior systems.

      • Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads [Paper]

        • NUS & USTC & UC Berkeley

    • KV Cache Management

      • DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants [Paper] [arXiv]

        • UChicago & Microsoft

        • Introduce DroidSpeak, the first distributed LLM inference system that reuses prefix KV caches across different LLMs with the same architecture, including across distributed nodes.

        • Selectively recompute a small subset of layers from another model's KV cache and reuse the remaining layers, then pipeline recomputation with reused-cache loading to improve performance while preserving quality.

        • Improve throughput by up to 4x and prefill latency by about 3.1x with negligible quality loss.

      • SYMPHONY: Improving Memory Management for LLM Inference Workloads [Paper] [arXiv]

        • UT Austin & UW-Madison

        • Exploit hints from multi-turn workloads to migrate KV caches off the critical serving path instead of recomputing them or pinning serving to specific machines through host-memory offload.

        • Dynamically migrate KV caches to enable fine-grained scheduling of inference requests across the cluster.

        • Handle more than 8x as many requests as state-of-the-art baselines while preserving a similar latency profile.

    • Workload Characterization

      • Seshat: Workload Characterization and Generation of Large Language Model Serving in Production [Paper] [arXiv]

        • PKU & Alibaba

        • Characterize production LLM serving workloads from a worldwide cloud inference service, covering language, multimodal, and reasoning models.

        • Build a per-client workload generation framework that composes realistic serving workloads from the observed production characteristics.

        • Avoid 50% under-provisioning compared with naive workload generation in a production validation case.

    • Serverless Computing

      • HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds [Paper] [arXiv] [Code]

        • PKU & Alibaba Cloud

        • Present HydraServe, a serverless LLM serving system for public clouds that minimizes cold-start latency.

        • Proactively distribute models across servers, overlap cold-start stages within workers, place workers to avoid GPU-network contention, and consolidate pipelines to reduce cold-start resource usage.

        • Reduce cold-start latency by 1.7x to 4.7x and improve SLO attainment by 1.43x to 1.74x over baselines.

    • Multiplexing

      • FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees [Paper] [arXiv]

        • CMU & Purdue & Anthropic & Mistral AI & Stanford

        • Introduce FlexLLM, the first system to co-serve LLM inference and parameter-efficient fine-tuning on shared GPUs by fusing computation at the token level.

        • Use dependent parallelization and graph pruning to shrink activation memory, then interleave inference and training tokens with token-level finetuning and a hybrid token scheduler to meet latency SLOs.

        • Save up to 80% of GPU memory and improve finetuning throughput by 1.9x to 4.8x under heavy inference load and 2.5x to 6.8x under light load.

    • LLM Agents

      • Agentix: An Efficient Serving Engine for LLM Agents as General Programs [Paper] [arXiv]

        • UC Berkeley & Google DeepMind & SJTU

        • Treat agent programs as first-class scheduling entities in LLM serving to reduce end-to-end latency for agentic workloads.

        • Intercept program-issued LLM calls to expose program-level context and preemptively prioritize calls based on previously completed work for both single-threaded and distributed programs.

        • Improve program throughput by 4x to 15x at the same latency compared with systems such as vLLM.

      • Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge Caching [Paper] [arXiv]

        • NUS & USTC & UofT & Sea AI Lab

        • Introduce Cortex, a cross-region knowledge caching architecture for LLM agents that targets semantic reuse rather than exact-match query reuse.

        • Build semantic-aware retrieval on Semantic Element and Semantic Retrieval Index abstractions, then add semantic-aware cache hits, eviction, prefetching, and a colocated lightweight LLM judger.

        • Increase throughput by up to 3.6x on search workloads while preserving accuracy close to uncached baselines, and improve coding-task throughput by 20%.

    • MoE Inference

      • SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading [Paper]

        • Tencent & Nanjing University & Zhongguancun Laboratory

  • LLM Fine-Tuning

    • MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing [Paper] [arXiv]

      • SJTU & NUS

      • Present MuxTune, a multi-tenant fine-tuning system that spatially and temporally multiplexes the shared backbone across concurrent parameter-efficient fine-tuning tasks.

      • Build on unified fine-tuning representations with hierarchical co-scheduling across tasks, operators, and data, including hybrid spatial-temporal multiplexing, two-tier hybrid parallelism, and chunk-based data alignment.

      • Achieve up to 2.33x higher throughput and 5.29x lower memory usage than prior baselines.

  • LLM Storage

    • ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression [Paper] [arXiv] [Homepage]

      • UVA & Harvard

      • Characterize all publicly available Hugging Face LLM repositories and identify structured sparse deltas within model families, bitwise-similarity-based family clustering, and tensor-level deduplication as the right storage granularity.

      • Design BitX, a lossless delta compression algorithm for XORed differences between fine-tuned and base models, and build ZipLLM to unify tensor-level deduplication with BitX compression.

      • Reduce model storage consumption by 54%, over 20% better than prior deduplication and compression approaches.

Distributed Training

  • Checkpointing

    • Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication [Paper] [arXiv]

      • Tufts & MIT

      • Introduce Checkmate, a model checkpointing system that reuses network-replicated gradients to avoid additional network transfer and disk I/O overhead on the normal checkpointing path.

      • Use dynamically reconfigurable in-network checkpoint replica placement and a failure-resilient accelerator pipeline to preserve resilience under failures while overlapping checkpoint creation with gradient computation and propagation.

      • Report nearly 100% throughput improvement over GPU-optimized checkpointing and up to 13.7% over asynchronous checkpointing on a 32-node GPU cluster.

  • Collective Communication

    • FAST: An Efficient Scheduler for All-to-All GPU Communication [Paper] [arXiv]

      • CMU & MangoBoost & UW & UPenn

      • Present FAST, an efficient All-to-All(v) scheduler for modern ML workloads, especially MoE models on heterogeneous two-tier GPU fabrics.

      • Address workload skew through intra-server rebalancing and enforce balanced one-to-one scale-out transfers to avoid incast congestion.

      • Outperform prior schedulers on skewed workloads while reducing schedule synthesis time by orders of magnitude.

    • HeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous GPU Clusters [Paper]

      • Northeastern University & Alibaba Cloud & SIAT, CAS

    • ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics [Paper] [arXiv]

      • UW & THU & Microsoft

      • Present ForestColl, a schedule generation tool that constructs broadcast and aggregation spanning trees to produce throughput-optimal collective communication schedules for arbitrary network topologies.

      • Achieve theoretical optimality with polynomial-time schedule generation while supporting both switching fabrics and direct accelerator interconnects.

      • Outperform vendor communication libraries and prior schedule generation methods on AMD MI250 and NVIDIA DGX A100 and H100 clusters, including LLM training workloads.

Acronyms

  • KV: Key-Value

  • LLM: Large Language Model

  • MoE: Mixture-of-Experts

  • RL: Reinforcement Learning

  • SLO: Service Level Objective

Last updated