SOSP 2026
Meta Info
Homepage: https://sigops.org/s/conferences/sosp/2026/
Paper List
Accepted Papers: https://sigops.org/s/conferences/sosp/2026/accepted.html
Proceedings: https://dl.acm.org/doi/proceedings/10.1145/3830418
Acceptance Rate
15.9% (= 62 / 390)
Papers
Large Language Models (LLMs)
LLM Training
Training Emulation
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with CrystalLLM [Paper]
Alibaba & SJTU & Harvard & ZJU
Constructs a scale-specific execution graph with computation, communication, and dependencies; runs selected ranks on physical GPUs while replaying other ranks as virtual participants.
Emulates up to 8192 GPUs with fewer than 1% of the original GPUs, reproducing iteration time with 0.58% average error.
RL Post-Training
TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training [Paper]
UW-Madison & ByteDance Seed
Introduces Reference-Oriented Storage (ROS), which tracks existing GPU-resident weight replicas and serves transfers directly from their holders instead of storing additional copies.
Combines topology-aware transfer, model-parallel consistency, and fault tolerance for elastic and cross-datacenter RL rollouts.
Fault Tolerance and Rescheduling
Anchor: Mitigating GPU Shallow Disruptions with Decoupled Memory [Paper]
THU & SJTU
Decouples GPU memory ownership from worker processes using a daemon and GPU IPC, allowing restarted workers to remap surviving tensors after process failures.
Adds tensor identity resolution and application-aware consistency protocols; integrates with DeepSpeed and vLLM to reduce recovery time for training and inference.
LLM Inference
Deterministic Inference
LLM-42: Enabling Determinism in LLM Inference with Verified Speculation [Paper]
MSR India & UW
Uses a nondeterministic decoding fast path and a fixed-batch verifier that commits matching tokens and rolls back mismatches.
Preserves dynamic batching and reuses existing GPU kernels, with overhead proportional to traffic requiring deterministic outputs.
Offline Inference
Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC [Paper]
SJTU & Alibaba
Characterizes 1.5 million offline tasks with 23 billion requests; exploits task determinism and throughput-oriented objectives in resource orchestration, caching, and execution.
Uses idle capacity in online prefill stages for offline tasks, improving hybrid-serving throughput by up to 35.7%.
MoE Inference
StreamEP: Straggler-Tolerant MoE Decoding without Communication Barriers [Paper]
USC & SNU & Google DeepMind & UT Austin
Removes layer-wide collective barriers by letting each GPU receive tokens, execute layers, and forward results independently.
Implements StreamInfer on NCCL with per-layer queues, a defragmenting scheduler, and variable-size asynchronous transfers for commodity GPU clusters.
Multi-LLM Serving
Janus: Multi-LLM Serving at Production Scale [Paper] [Code]
PKU & JD & UCAS & USTC & SJTU
Co-designs service and engine layers with dual-timescale model scheduling, combining steady-state colocation with sub-second elastic scaling.
Uses HBM virtualization, device-to-device model forks, and an SLO-oriented request scheduler to handle production bursts across heterogeneous models.
Workload Characterization
LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems [Paper]
Rank Computing & THU
Embeds lightweight probe kernels as trace nodes in GPU execution graphs and streams buffered profiling data to CPUs without interrupting user kernels.
Profiles production training, serving, and fine-tuning with 0.9% average overhead; uncovered 15 performance issues during eight months of deployment.
End-to-End Serving
Taming Inference Workloads at Global Scale: Foundation Model Serving in Amazon Bedrock [Paper]
AWS
Describes Bedrock's workload management across admission control, classification, cross-region routing, scheduling, and orchestration.
Decouples ingress from execution with cache-aware routing; uses bounded-staleness local state and periodic quota reconciliation to avoid synchronous global coordination.
LLM Applications
Agentic Systems
Skill VM: Write Once, Run Everywhere Efficiently [Paper] [arXiv] [Code] [Homepage]
SJTU IPADS
Introduces SkillVM, treating skills as programs and model-harness pairs as heterogeneous processors to improve portability across agent runtimes.
Combines capability-based compilation, environment binding, and concurrency extraction with runtime JIT code solidification and adaptive recompilation.
Evaluates across ten LLMs and four harnesses; reduces token consumption by up to 40%, improves parallel execution by up to 3.2x, and cuts latency by 19-50x through code solidification.
Agent Data Infrastructure
AgileLog: A Forkable Shared Log for Agents on Data Streams [Paper]
UIUC
Extends the shared-log abstraction with forks so agents can act on streaming data without interfering with traditional streaming tasks.
Implements Bolt with inexpensive forks and logical and performance isolation to support agentic writes.
Don't Let AI Agents YOLO Your Files: Information and Control in Agent-Native Filesystems [Paper]
UW-Madison & MSR & Iowa State University
Studies 290 reports of agent filesystem misuse and proposes primitives to inspect effects, undo mutations, and gate access.
Builds YoloFS with staged mutations, intermediate snapshots, and progressive permissions for agent self-correction and user review.
Voice Applications
Scalable Context Orchestration for Serving LLMs Over Voice [Paper]
SJTU & AgenticSys
Introduces llmovoice, a middleware that constructs bounded context from current input, relevant history, and explicit paralinguistic and environmental state.
Uses LLM-generated runtime directives to adapt spoken responses and reduce model usage cost during long voice sessions.
Diffusion Models
Diffusion Model Serving
DiFlow: A System for Micro-Serving Text-to-image Diffusion Workflows [Paper] [arXiv] [Code]
HKUST & Alibaba & SJTU
Our work!
Decomposes text-to-image diffusion workflows into independently managed model-execution nodes.
Enables per-model scaling, cross-workflow model sharing, and adaptive model parallelism for bursty serving workloads.
Model Serving
Multi-Model Serving
Sandhi: Fine-Grained Merging for Memory Efficient Multi-Model Serving [Paper]
GaTech & Microsoft
Adaptively merges colocated fine-tuned models at component granularity while respecting user-specified accuracy requirements.
Reduces GPU memory footprint by up to 49.8%, enabling higher serving throughput and lower deployment cost.
Trusted Inference
Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference [Paper] [Code]
CMU & UT Austin & NVIDIA
Combines a DSL, optimizing compiler, and runtime for fully homomorphic encrypted inference on multiple GPUs.
Generates GPU kernels, compresses plaintext data, and manages terabyte-scale memory and cross-device communication for encrypted LLM inference.
Deep Learning Compilation
Agent-Based Kernel Generation
Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants [Paper] [arXiv] [Code]
CausalFlow & HKUST & UCAS & UC Riverside & THU & Stanford
Uses data-flow invariants as compile-time constraints for agentic GPU optimization through a tile-based DSL.
Propagates symbolic tags and checks assertions with SMT-backed analysis, returning violation witnesses to guide repairs without runtime checking overhead.
GPU Kernels and Compilers
Taming Dynamism on GPUs: Cross-SM Kernel Fusion via SM Cooperation and Just-in-Time Reduction [Paper]
SJTU IPADS
Introduces MeldKernel, targeting dynamic GPU workloads such as decoding attention and MoE expert routing.
Decomposes phase-level dependencies into sub-task dependencies with on-chip SM cooperation and asynchronous just-in-time reduction, reducing intermediate off-chip memory traffic.
Separates declarative computation from runtime hardware mapping and inter-SM cooperation; improves critical workloads by 1.3x on average over the best solutions.
Efficient GPU Multitasking with Morphable Kernels [Paper]
THU
Introduces MorphX, which transforms kernels at compile time so they can yield or reclaim compute resources without additional launches.
Uses in-place profiling to adapt scheduling to contention and increase background throughput under foreground latency constraints.
Kernel Verification
M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification [Paper]
Penn State & ICT, CAS & PKU
Uses HFProbe to trace model execution without GPUs and derive symbolic constraints separating model-fixed and user-variable kernel arguments.
Applies cuK-LEE symbolic execution under those constraints to detect CUDA memory bugs across dynamic tensor shapes and launch configurations.
Wafer-Scale Compilation and Runtime
MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference [Paper]
Edinburgh & MSR
Compiles dynamic inference events, including batching and MoE, into per-core schedules for asynchronous, decentralized execution.
Preserves low inference latency while increasing wafer-scale throughput by up to 56x over the prior system.
Wavel: A Fast and Efficient Compilation System for Wafer-Scale Accelerators [Paper]
Edinburgh & MSR
Introduces MeshIR to explicitly represent executable schedules covering placement, layout alignment, and execution order.
Searches schedules through locality-preserving CUT and TUNE primitives with physical cost evaluation and structured pruning.
Resource Management
GPU Scheduling and Sharing
Beyond Utilization: Energy-Conscious GPU Sharing for Inference Serving [Paper]
UT Austin
Introduces EnerTune, which models performance and power for individual and colocated models on shared GPUs.
Uses energy-aware bin packing to jointly choose placement, GPU allocation, frequency, and batch size while meeting serving SLOs.
Linux AGX: An Adaptive GPU eXtension to Linux Fair Scheduling for Physical AI and Robotic Systems [Paper]
UC Riverside
Makes Linux fair-scheduling weights aware of GPU launch dependencies and evolving critical paths in robotic task graphs.
Combines a user-space importance profiler with bounded kernel weight adjustments while preserving proportional fairness.
Model Storage and Data Pipeline
Model Storage
TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models [Paper]
UVA & Harvard
Applies lossless delta compression at tensor granularity instead of pairing whole models.
Uses bit-level fingerprints and incremental multi-center clustering to select compression bases without reliable model lineage metadata.
ML for Systems
System Optimization
TuxBot: Semantic-Aware Online OS Tuning with LLMs [Paper]
Columbia & IBM Research
Builds an OS-tuning context from telemetry, parameter schemas, tuning trajectories, and prior sessions, pairing low-latency and deeper reasoning models.
Applies typed, validated parameter updates to tune live Linux services, including settings without direct application-level performance metrics.
Acronyms
AGX: Adaptive GPU eXtension
AI: Artificial Intelligence
CPU: Central Processing Unit
CUDA: Compute Unified Device Architecture
DSL: Domain-Specific Language
GPU: Graphics Processing Unit
HBM: High Bandwidth Memory
IPC: Inter-Process Communication
JIT: Just-in-Time
LLM: Large Language Model
ML: Machine Learning
MoE: Mixture-of-Experts
NCCL: NVIDIA Collective Communication Library
OS: Operating System
RL: Reinforcement Learning
ROS: Reference-Oriented Storage
SLO: Service Level Objective
SM: Streaming Multiprocessor
SMT: Satisfiability Modulo Theories
VM: Virtual Machine
Last updated