> For the complete documentation index, see [llms.txt](https://paper.lingyunyang.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://paper.lingyunyang.com/reading-notes/conference/sosp-2026.md).

# SOSP 2026

## Meta Info

Homepage: <https://sigops.org/s/conferences/sosp/2026/>

### Paper List

* Accepted Papers: <https://sigops.org/s/conferences/sosp/2026/accepted.html>
* Proceedings: <https://dl.acm.org/doi/proceedings/10.1145/3830418>

### Acceptance Rate

15.9% (= 62 / 390)

## Papers

### Large Language Models (LLMs)

* LLM Training
  * Training Emulation
    * A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with CrystalLLM \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843852)]
      * Alibaba & SJTU & Harvard & ZJU
      * Constructs a scale-specific execution graph with computation, communication, and dependencies; runs selected ranks on physical GPUs while replaying other ranks as virtual participants.
      * Emulates up to 8192 GPUs with fewer than 1% of the original GPUs, reproducing iteration time with 0.58% average error.
  * RL Post-Training
    * TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843859)]
      * UW-Madison & ByteDance Seed
      * Introduces Reference-Oriented Storage (ROS), which tracks existing GPU-resident weight replicas and serves transfers directly from their holders instead of storing additional copies.
      * Combines topology-aware transfer, model-parallel consistency, and fault tolerance for elastic and cross-datacenter RL rollouts.
  * Fault Tolerance and Rescheduling
    * Anchor: Mitigating GPU Shallow Disruptions with Decoupled Memory \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843892)]
      * THU & SJTU
      * Decouples GPU memory ownership from worker processes using a daemon and GPU IPC, allowing restarted workers to remap surviving tensors after process failures.
      * Adds tensor identity resolution and application-aware consistency protocols; integrates with DeepSpeed and vLLM to reduce recovery time for training and inference.
* LLM Inference
  * Deterministic Inference
    * LLM-42: Enabling Determinism in LLM Inference with Verified Speculation \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843901)]
      * MSR India & UW
      * Uses a nondeterministic decoding fast path and a fixed-batch verifier that commits matching tokens and rolls back mismatches.
      * Preserves dynamic batching and reuses existing GPU kernels, with overhead proportional to traffic requiring deterministic outputs.
  * Offline Inference
    * Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843877)]
      * SJTU & Alibaba
      * Characterizes 1.5 million offline tasks with 23 billion requests; exploits task determinism and throughput-oriented objectives in resource orchestration, caching, and execution.
      * Uses idle capacity in online prefill stages for offline tasks, improving hybrid-serving throughput by up to 35.7%.
  * MoE Inference
    * StreamEP: Straggler-Tolerant MoE Decoding without Communication Barriers \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843869)]
      * USC & SNU & Google DeepMind & UT Austin
      * Removes layer-wide collective barriers by letting each GPU receive tokens, execute layers, and forward results independently.
      * Implements StreamInfer on NCCL with per-layer queues, a defragmenting scheduler, and variable-size asynchronous transfers for commodity GPU clusters.
  * Multi-LLM Serving
    * Janus: Multi-LLM Serving at Production Scale \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843867)] \[[Code](https://github.com/Janus2026/Janus)]
      * PKU & JD & UCAS & USTC & SJTU
      * Co-designs service and engine layers with dual-timescale model scheduling, combining steady-state colocation with sub-second elastic scaling.
      * Uses HBM virtualization, device-to-device model forks, and an SLO-oriented request scheduler to handle production bursts across heterogeneous models.
  * Workload Characterization
    * LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843871)]
      * Rank Computing & THU
      * Embeds lightweight probe kernels as trace nodes in GPU execution graphs and streams buffered profiling data to CPUs without interrupting user kernels.
      * Profiles production training, serving, and fine-tuning with 0.9% average overhead; uncovered 15 performance issues during eight months of deployment.
  * End-to-End Serving
    * Taming Inference Workloads at Global Scale: Foundation Model Serving in Amazon Bedrock \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843855)]
      * AWS
      * Describes Bedrock's workload management across admission control, classification, cross-region routing, scheduling, and orchestration.
      * Decouples ingress from execution with cache-aware routing; uses bounded-staleness local state and periodic quota reconciliation to avoid synchronous global coordination.
* LLM Applications
  * Agentic Systems
    * Skill VM: Write Once, Run Everywhere Efficiently \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843874)] \[[arXiv](https://arxiv.org/abs/2604.03088)] \[[Code](https://github.com/SJTU-IPADS/SkVM/)] \[[Homepage](https://skillvm.ai/)]
      * SJTU IPADS
      * Introduces **SkillVM**, treating skills as programs and model-harness pairs as heterogeneous processors to improve portability across agent runtimes.
      * Combines capability-based compilation, environment binding, and concurrency extraction with runtime JIT code solidification and adaptive recompilation.
      * Evaluates across ten LLMs and four harnesses; reduces token consumption by up to 40%, improves parallel execution by up to 3.2x, and cuts latency by 19-50x through code solidification.
  * Agent Data Infrastructure
    * AgileLog: A Forkable Shared Log for Agents on Data Streams \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843875)]
      * UIUC
      * Extends the shared-log abstraction with forks so agents can act on streaming data without interfering with traditional streaming tasks.
      * Implements Bolt with inexpensive forks and logical and performance isolation to support agentic writes.
    * Don't Let AI Agents YOLO Your Files: Information and Control in Agent-Native Filesystems \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843858)]
      * UW-Madison & MSR & Iowa State University
      * Studies 290 reports of agent filesystem misuse and proposes primitives to inspect effects, undo mutations, and gate access.
      * Builds YoloFS with staged mutations, intermediate snapshots, and progressive permissions for agent self-correction and user review.
  * Voice Applications
    * Scalable Context Orchestration for Serving LLMs Over Voice \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843860)]
      * SJTU & AgenticSys
      * Introduces **llmovoice**, a middleware that constructs bounded context from current input, relevant history, and explicit paralinguistic and environmental state.
      * Uses LLM-generated runtime directives to adapt spoken responses and reduce model usage cost during long voice sessions.

### Diffusion Models

* Diffusion Model Serving
  * DiFlow: A System for Micro-Serving Text-to-image Diffusion Workflows \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843880)] \[[arXiv](https://arxiv.org/abs/2604.08123)] \[[Code](https://github.com/diflow-project/diflow)]
    * HKUST & Alibaba & SJTU
    * **Our work!**
    * Decomposes text-to-image diffusion workflows into independently managed model-execution nodes.
    * Enables per-model scaling, cross-workflow model sharing, and adaptive model parallelism for bursty serving workloads.

### Model Serving

* Multi-Model Serving
  * Sandhi: Fine-Grained Merging for Memory Efficient Multi-Model Serving \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843911)]
    * GaTech & Microsoft
    * Adaptively merges colocated fine-tuned models at component granularity while respecting user-specified accuracy requirements.
    * Reduces GPU memory footprint by up to 49.8%, enabling higher serving throughput and lower deployment cost.
* Trusted Inference
  * Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843864)] \[[Code](https://github.com/CMU-CAOS/Cerium)]
    * CMU & UT Austin & NVIDIA
    * Combines a DSL, optimizing compiler, and runtime for fully homomorphic encrypted inference on multiple GPUs.
    * Generates GPU kernels, compresses plaintext data, and manages terabyte-scale memory and cross-device communication for encrypted LLM inference.

### Deep Learning Compilation

* Agent-Based Kernel Generation
  * Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843902)] \[[arXiv](https://arxiv.org/abs/2604.18616)] \[[Code](https://github.com/causalflowai/avelang)]
    * CausalFlow & HKUST & UCAS & UC Riverside & THU & Stanford
    * Uses data-flow invariants as compile-time constraints for agentic GPU optimization through a tile-based DSL.
    * Propagates symbolic tags and checks assertions with SMT-backed analysis, returning violation witnesses to guide repairs without runtime checking overhead.
* GPU Kernels and Compilers
  * Taming Dynamism on GPUs: Cross-SM Kernel Fusion via SM Cooperation and Just-in-Time Reduction \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843857)]
    * SJTU IPADS
    * Introduces **MeldKernel**, targeting dynamic GPU workloads such as decoding attention and MoE expert routing.
    * Decomposes phase-level dependencies into sub-task dependencies with on-chip SM cooperation and asynchronous just-in-time reduction, reducing intermediate off-chip memory traffic.
    * Separates declarative computation from runtime hardware mapping and inter-SM cooperation; improves critical workloads by 1.3x on average over the best solutions.
  * Efficient GPU Multitasking with Morphable Kernels \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843904)]
    * THU
    * Introduces **MorphX**, which transforms kernels at compile time so they can yield or reclaim compute resources without additional launches.
    * Uses in-place profiling to adapt scheduling to contention and increase background throughput under foreground latency constraints.
* Kernel Verification
  * M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843908)]
    * Penn State & ICT, CAS & PKU
    * Uses HFProbe to trace model execution without GPUs and derive symbolic constraints separating model-fixed and user-variable kernel arguments.
    * Applies cuK-LEE symbolic execution under those constraints to detect CUDA memory bugs across dynamic tensor shapes and launch configurations.
* Wafer-Scale Compilation and Runtime
  * MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843856)]
    * Edinburgh & MSR
    * Compiles dynamic inference events, including batching and MoE, into per-core schedules for asynchronous, decentralized execution.
    * Preserves low inference latency while increasing wafer-scale throughput by up to 56x over the prior system.
  * Wavel: A Fast and Efficient Compilation System for Wafer-Scale Accelerators \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843884)]
    * Edinburgh & MSR
    * Introduces MeshIR to explicitly represent executable schedules covering placement, layout alignment, and execution order.
    * Searches schedules through locality-preserving CUT and TUNE primitives with physical cost evaluation and structured pruning.

### Resource Management

* GPU Scheduling and Sharing
  * Beyond Utilization: Energy-Conscious GPU Sharing for Inference Serving \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843913)]
    * UT Austin
    * Introduces **EnerTune**, which models performance and power for individual and colocated models on shared GPUs.
    * Uses energy-aware bin packing to jointly choose placement, GPU allocation, frequency, and batch size while meeting serving SLOs.
  * Linux AGX: An Adaptive GPU eXtension to Linux Fair Scheduling for Physical AI and Robotic Systems \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843893)]
    * UC Riverside
    * Makes Linux fair-scheduling weights aware of GPU launch dependencies and evolving critical paths in robotic task graphs.
    * Combines a user-space importance profiler with bounded kernel weight adjustments while preserving proportional fairness.

### Model Storage and Data Pipeline

* Model Storage
  * TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843872)]
    * UVA & Harvard
    * Applies lossless delta compression at tensor granularity instead of pairing whole models.
    * Uses bit-level fingerprints and incremental multi-center clustering to select compression bases without reliable model lineage metadata.

### ML for Systems

* System Optimization
  * TuxBot: Semantic-Aware Online OS Tuning with LLMs \[[Paper](https://dl.acm.org/doi/10.1145/3830418.3843885)]
    * Columbia & IBM Research
    * Builds an OS-tuning context from telemetry, parameter schemas, tuning trajectories, and prior sessions, pairing low-latency and deeper reasoning models.
    * Applies typed, validated parameter updates to tune live Linux services, including settings without direct application-level performance metrics.

## Acronyms

* AGX: Adaptive GPU eXtension
* AI: Artificial Intelligence
* CPU: Central Processing Unit
* CUDA: Compute Unified Device Architecture
* DSL: Domain-Specific Language
* GPU: Graphics Processing Unit
* HBM: High Bandwidth Memory
* IPC: Inter-Process Communication
* JIT: Just-in-Time
* LLM: Large Language Model
* ML: Machine Learning
* MoE: Mixture-of-Experts
* NCCL: NVIDIA Collective Communication Library
* OS: Operating System
* RL: Reinforcement Learning
* ROS: Reference-Oriented Storage
* SLO: Service Level Objective
* SM: Streaming Multiprocessor
* SMT: Satisfiability Modulo Theories
* VM: Virtual Machine