Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

SOSP 2026

Meta Info

Homepage: https://sigops.org/s/conferences/sosp/2026/

Paper List

Acceptance Rate

15.9% (= 62 / 390)

Papers

Large Language Models (LLMs)

  • LLM Training

    • Training Emulation

      • A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with CrystalLLM [Paper]

        • Alibaba & SJTU & Harvard & ZJU

        • Constructs a scale-specific execution graph with computation, communication, and dependencies; runs selected ranks on physical GPUs while replaying other ranks as virtual participants.

        • Emulates up to 8192 GPUs with fewer than 1% of the original GPUs, reproducing iteration time with 0.58% average error.

    • RL Post-Training

      • TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training [Paper]

        • UW-Madison & ByteDance Seed

        • Introduces Reference-Oriented Storage (ROS), which tracks existing GPU-resident weight replicas and serves transfers directly from their holders instead of storing additional copies.

        • Combines topology-aware transfer, model-parallel consistency, and fault tolerance for elastic and cross-datacenter RL rollouts.

    • Fault Tolerance and Rescheduling

      • Anchor: Mitigating GPU Shallow Disruptions with Decoupled Memory [Paper]

        • THU & SJTU

        • Decouples GPU memory ownership from worker processes using a daemon and GPU IPC, allowing restarted workers to remap surviving tensors after process failures.

        • Adds tensor identity resolution and application-aware consistency protocols; integrates with DeepSpeed and vLLM to reduce recovery time for training and inference.

  • LLM Inference

    • Deterministic Inference

      • LLM-42: Enabling Determinism in LLM Inference with Verified Speculation [Paper]

        • MSR India & UW

        • Uses a nondeterministic decoding fast path and a fixed-batch verifier that commits matching tokens and rolls back mismatches.

        • Preserves dynamic batching and reuses existing GPU kernels, with overhead proportional to traffic requiring deterministic outputs.

    • Offline Inference

      • Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC [Paper]

        • SJTU & Alibaba

        • Characterizes 1.5 million offline tasks with 23 billion requests; exploits task determinism and throughput-oriented objectives in resource orchestration, caching, and execution.

        • Uses idle capacity in online prefill stages for offline tasks, improving hybrid-serving throughput by up to 35.7%.

    • MoE Inference

      • StreamEP: Straggler-Tolerant MoE Decoding without Communication Barriers [Paper]

        • USC & SNU & Google DeepMind & UT Austin

        • Removes layer-wide collective barriers by letting each GPU receive tokens, execute layers, and forward results independently.

        • Implements StreamInfer on NCCL with per-layer queues, a defragmenting scheduler, and variable-size asynchronous transfers for commodity GPU clusters.

    • Multi-LLM Serving

      • Janus: Multi-LLM Serving at Production Scale [Paper] [Code]

        • PKU & JD & UCAS & USTC & SJTU

        • Co-designs service and engine layers with dual-timescale model scheduling, combining steady-state colocation with sub-second elastic scaling.

        • Uses HBM virtualization, device-to-device model forks, and an SLO-oriented request scheduler to handle production bursts across heterogeneous models.

    • Workload Characterization

      • LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems [Paper]

        • Rank Computing & THU

        • Embeds lightweight probe kernels as trace nodes in GPU execution graphs and streams buffered profiling data to CPUs without interrupting user kernels.

        • Profiles production training, serving, and fine-tuning with 0.9% average overhead; uncovered 15 performance issues during eight months of deployment.

    • End-to-End Serving

      • Taming Inference Workloads at Global Scale: Foundation Model Serving in Amazon Bedrock [Paper]

        • AWS

        • Describes Bedrock's workload management across admission control, classification, cross-region routing, scheduling, and orchestration.

        • Decouples ingress from execution with cache-aware routing; uses bounded-staleness local state and periodic quota reconciliation to avoid synchronous global coordination.

  • LLM Applications

    • Agentic Systems

      • Skill VM: Write Once, Run Everywhere Efficiently [Paper] [arXiv] [Code] [Homepage]

        • SJTU IPADS

        • Introduces SkillVM, treating skills as programs and model-harness pairs as heterogeneous processors to improve portability across agent runtimes.

        • Combines capability-based compilation, environment binding, and concurrency extraction with runtime JIT code solidification and adaptive recompilation.

        • Evaluates across ten LLMs and four harnesses; reduces token consumption by up to 40%, improves parallel execution by up to 3.2x, and cuts latency by 19-50x through code solidification.

    • Agent Data Infrastructure

      • AgileLog: A Forkable Shared Log for Agents on Data Streams [Paper]

        • UIUC

        • Extends the shared-log abstraction with forks so agents can act on streaming data without interfering with traditional streaming tasks.

        • Implements Bolt with inexpensive forks and logical and performance isolation to support agentic writes.

      • Don't Let AI Agents YOLO Your Files: Information and Control in Agent-Native Filesystems [Paper]

        • UW-Madison & MSR & Iowa State University

        • Studies 290 reports of agent filesystem misuse and proposes primitives to inspect effects, undo mutations, and gate access.

        • Builds YoloFS with staged mutations, intermediate snapshots, and progressive permissions for agent self-correction and user review.

    • Voice Applications

      • Scalable Context Orchestration for Serving LLMs Over Voice [Paper]

        • SJTU & AgenticSys

        • Introduces llmovoice, a middleware that constructs bounded context from current input, relevant history, and explicit paralinguistic and environmental state.

        • Uses LLM-generated runtime directives to adapt spoken responses and reduce model usage cost during long voice sessions.

Diffusion Models

  • Diffusion Model Serving

    • DiFlow: A System for Micro-Serving Text-to-image Diffusion Workflows [Paper] [arXiv] [Code]

      • HKUST & Alibaba & SJTU

      • Our work!

      • Decomposes text-to-image diffusion workflows into independently managed model-execution nodes.

      • Enables per-model scaling, cross-workflow model sharing, and adaptive model parallelism for bursty serving workloads.

Model Serving

  • Multi-Model Serving

    • Sandhi: Fine-Grained Merging for Memory Efficient Multi-Model Serving [Paper]

      • GaTech & Microsoft

      • Adaptively merges colocated fine-tuned models at component granularity while respecting user-specified accuracy requirements.

      • Reduces GPU memory footprint by up to 49.8%, enabling higher serving throughput and lower deployment cost.

  • Trusted Inference

    • Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference [Paper] [Code]

      • CMU & UT Austin & NVIDIA

      • Combines a DSL, optimizing compiler, and runtime for fully homomorphic encrypted inference on multiple GPUs.

      • Generates GPU kernels, compresses plaintext data, and manages terabyte-scale memory and cross-device communication for encrypted LLM inference.

Deep Learning Compilation

  • Agent-Based Kernel Generation

    • Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants [Paper] [arXiv] [Code]

      • CausalFlow & HKUST & UCAS & UC Riverside & THU & Stanford

      • Uses data-flow invariants as compile-time constraints for agentic GPU optimization through a tile-based DSL.

      • Propagates symbolic tags and checks assertions with SMT-backed analysis, returning violation witnesses to guide repairs without runtime checking overhead.

  • GPU Kernels and Compilers

    • Taming Dynamism on GPUs: Cross-SM Kernel Fusion via SM Cooperation and Just-in-Time Reduction [Paper]

      • SJTU IPADS

      • Introduces MeldKernel, targeting dynamic GPU workloads such as decoding attention and MoE expert routing.

      • Decomposes phase-level dependencies into sub-task dependencies with on-chip SM cooperation and asynchronous just-in-time reduction, reducing intermediate off-chip memory traffic.

      • Separates declarative computation from runtime hardware mapping and inter-SM cooperation; improves critical workloads by 1.3x on average over the best solutions.

    • Efficient GPU Multitasking with Morphable Kernels [Paper]

      • THU

      • Introduces MorphX, which transforms kernels at compile time so they can yield or reclaim compute resources without additional launches.

      • Uses in-place profiling to adapt scheduling to contention and increase background throughput under foreground latency constraints.

  • Kernel Verification

    • M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification [Paper]

      • Penn State & ICT, CAS & PKU

      • Uses HFProbe to trace model execution without GPUs and derive symbolic constraints separating model-fixed and user-variable kernel arguments.

      • Applies cuK-LEE symbolic execution under those constraints to detect CUDA memory bugs across dynamic tensor shapes and launch configurations.

  • Wafer-Scale Compilation and Runtime

    • MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference [Paper]

      • Edinburgh & MSR

      • Compiles dynamic inference events, including batching and MoE, into per-core schedules for asynchronous, decentralized execution.

      • Preserves low inference latency while increasing wafer-scale throughput by up to 56x over the prior system.

    • Wavel: A Fast and Efficient Compilation System for Wafer-Scale Accelerators [Paper]

      • Edinburgh & MSR

      • Introduces MeshIR to explicitly represent executable schedules covering placement, layout alignment, and execution order.

      • Searches schedules through locality-preserving CUT and TUNE primitives with physical cost evaluation and structured pruning.

Resource Management

  • GPU Scheduling and Sharing

    • Beyond Utilization: Energy-Conscious GPU Sharing for Inference Serving [Paper]

      • UT Austin

      • Introduces EnerTune, which models performance and power for individual and colocated models on shared GPUs.

      • Uses energy-aware bin packing to jointly choose placement, GPU allocation, frequency, and batch size while meeting serving SLOs.

    • Linux AGX: An Adaptive GPU eXtension to Linux Fair Scheduling for Physical AI and Robotic Systems [Paper]

      • UC Riverside

      • Makes Linux fair-scheduling weights aware of GPU launch dependencies and evolving critical paths in robotic task graphs.

      • Combines a user-space importance profiler with bounded kernel weight adjustments while preserving proportional fairness.

Model Storage and Data Pipeline

  • Model Storage

    • TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models [Paper]

      • UVA & Harvard

      • Applies lossless delta compression at tensor granularity instead of pairing whole models.

      • Uses bit-level fingerprints and incremental multi-center clustering to select compression bases without reliable model lineage metadata.

ML for Systems

  • System Optimization

    • TuxBot: Semantic-Aware Online OS Tuning with LLMs [Paper]

      • Columbia & IBM Research

      • Builds an OS-tuning context from telemetry, parameter schemas, tuning trajectories, and prior sessions, pairing low-latency and deeper reasoning models.

      • Applies typed, validated parameter updates to tune live Linux services, including settings without direct application-level performance metrics.

Acronyms

  • AGX: Adaptive GPU eXtension

  • AI: Artificial Intelligence

  • CPU: Central Processing Unit

  • CUDA: Compute Unified Device Architecture

  • DSL: Domain-Specific Language

  • GPU: Graphics Processing Unit

  • HBM: High Bandwidth Memory

  • IPC: Inter-Process Communication

  • JIT: Just-in-Time

  • LLM: Large Language Model

  • ML: Machine Learning

  • MoE: Mixture-of-Experts

  • NCCL: NVIDIA Collective Communication Library

  • OS: Operating System

  • RL: Reinforcement Learning

  • ROS: Reference-Oriented Storage

  • SLO: Service Level Objective

  • SM: Streaming Multiprocessor

  • SMT: Satisfiability Modulo Theories

  • VM: Virtual Machine

Last updated