ATC 2024

Meta Info

Homepage: https://www.usenix.org/conference/atc24

Paper list: https://www.usenix.org/conference/atc24/technical-sessions

Acceptance Rate

15.8% (= 77 / 488)

Papers

Large Language Models (LLMs)

• Serving LLMs

  • Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention [Paper]

    • NUS & SJTU & Huawei Cloud

    • Reuse KV caches across multi-turn conversations; maintain a hierarchical KV caching system; layer-wise pre-loading and asynchronous saving; scheduler-aware fetching and eviction.

  • Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs [Paper] [Code]

    • Sydney & Microsoft & Rutgers

    • TC-FPx, the first full-stack GPU kernel design scheme with unified Tensor Core support of 6-bit and arbitrary bit-width quantization (e.g., 5-bit).

• LLM alignment / RLHF training

  • PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch [Paper]

    • THU

    • Intra-stage switching: explore model affinities and overlap computation via time-sharing.

    • Inter-stage switching: find the optimal switch plan with the minimum communication cost.

    • Based on Megatron-LM.

• LLM federated fine-tuning

  • FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences [Paper] [Code]

    • BUPT

    • Employ backpropagation (BP)-free training methods, requiring devices only to execute “perturbed inferences”; adaptively allocate computational loads across devices to balance between convergence speed and accuracy.

• LLM training

  • Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Parallelism [Paper] [Code]

    • Kuaishou

    • The balance between computation and memory utilization.

    • Two activation rematerialization strategies

      • Pipeline-parallel-aware offloading to maximize the utilization of host memory for storing activations.

      • Compute-memory balanced checkpointing to balance between activation memory and computational efficiency.

Reliability

• AI Infra

  • SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation [Paper] [Code]

    • MSR & Microsoft

      • Best Paper Award

      • SuperBench, a proactive validation system for AI infrastructure that mitigates hidden degradation (i.e., gray failure) caused by hardware redundancies and enhances overall reliability.

      • A comprehensive benchmark suite to evaluate individual hardware components and represent most real AI workloads.

• HBM

  • Removing Obstacles before Breaking Through the Memory Wall: A Close Look at HBM Errors in the Field [Paper] [Code]

    • Xiamen University & Huawei & Minjiang University

      • Conduct the first systematical study on HBM errors, which cover over 460 million error events collected from nineteen data centers and span over two years of deployment under a variety of services.

      • Calchas, a hierarchical failure prediction framework for HBM integrates spatial, temporal, and sensor information from various device levels to predict upcoming failures.

Supercomputer

• Full Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer: What Can We Learn from It?

  • THU & SDU & National Supercomputer Center in Wuxi

  • A comprehensive analysis of six years’ worth of 40 TB data (comprising I/O performance data and job running information) from Sunway TaihuLight, boasting 41508 nodes.

  • Notice: The data is currently not available.

Distributed Training

• Metis: Fast Automatic Distributed Training on Heterogeneous GPUs [Paper]

  • Samsung Research & UNIST

  • Metis, a system to automatically finds efficient parallelism plans for distributed training on heterogeneous GPUs.

  • Balance loads with heterogeneity-awareness; prefer data parallelism over tensor parallelism within a pipeline stage.

  • Evaluated with three large models (GPT-3, MoE, and Wide-Resnet).

Data Preprocessing

• Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement [Paper] [Code]

  • ETH & Google

  • Dynamically schedule data preprocessing workers on ML accelerator host resources to minimize the number of remote CPU workers needed to achieve peak data ingestion bandwidth.

  • Analyze the characteristics of input pipelines and automatically reorder transformations to increase data preprocessing worker throughput

Serverless Computing

• Harmonizing Efficiency and Practicability: Optimizing Resource Utilization in Serverless Computing with Jiagu [Paper]

  • SJTU, IPAPS & Huawei Cloud & EPFL

  • Jiagu, a serverless system based on OpenFaaS

    • Pre-decision scheduling: decouple prediction and decision-making; predict every function's capacities on a server using a model.

    • Dual-staged scaling: frequent adjustment of instances.

• ALPS: An Adaptive Learning, Priority OS Scheduler for Serverless Functions [Paper] [Code]

  • UVA & George Mason University & Adobe Research

  • ALPS: Adaptive Learning, Priority Scheduler

    • Application-aware kernel scheduler

    • Frontend: user-space; approximate shortest remaining process time (SRPT) priority scheduling by adaptively learning from an SRPT simulation on recent past workload.

    • Backend: use eBPF functions hooked to CFS to inform scheduling decisions (from the frontend) in the kernel.

• StreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow [Paper] [Code]

  • HUST & INRIA

  • One GPU runtime per inference workflow instead of one GPU runtime per function.

  • Use CUDA streams for serverless inference; fine-grained GPU memory management; and PCIe bandwidth sharing among concurrent streams.

• A Secure, Fast, and Resource-Efficient Serverless Platform with Function REWIND [Paper] [Code]

  • Sungkyunkwan University & Yonsei University & Seoul National University

  • Enhance performance while maintaining strict data isolation between requests.

  • The container is reset to an initial state free of any sensitive data after each function request; incorporate a kernel-level memory snapshot management system; optimize runtime by reusing memory regions and leveraging the temporal locality of function executions.

Model Serving

• Power-aware Deep Learning Model Serving with μ-Serve [Paper]

  • UIUC & IBM Research

  • Scaling GPU frequency for power saving without SLO attainment violations.

Cluster Scheduler

• Starburst: A Cost-aware Scheduler for Hybrid Cloud [Paper] [Code]

  • UC Berkeley & UCSB

  • Distinguished Artifact Award

  • Run the batch workloads on the private clusters or public cloud. -> Trade-off between the cost and JCT.

  • Dynamically control jobs' waiting times to improve utilization.

    • Assign longer waits for large jobs to increase their chances of running on the cluster.

    • Assign shorter waits to small jobs to increase their chances of running on the cloud.

Deep Learning Compiler

• MAGPY: Compiling Eager Mode DNN Programs by Monitoring Execution States [Paper] [Code]

  • THU

  • Generate more complete operator graphs by collecting key runtime information through monitoring program execution.

  • Provide a reference graph to record program execution states and leverage reference relationships to identify state changes that can impact program outputs.

Deep Learning Recommendation Models (DLRMs)

• OPER: Optimality-Guided Embedding Table Parallelization for Large-scale Recommendation Model [Paper]

  • UCSD & UCSB & Meta & Pacific Northwest National Laboratory

  • Provide a near-optimal parallelization strategy for embedding tables.

Probabilistic Graphical Models

• Fast Inference for Probabilistic Graphical Models [Paper] [Code]

  • University of Western Australia & HKUST

  • Fast-PGM: a fast and parallel PGM inference system for importance sampling-based approximate inference algorithms.

Remote Direct Memory Access (RDMA)

• PeRF: Preemption-enabled RDMA Framework [Paper]

  • Acryl Inc. & Sungkyunkwan University

  • Offer software-based performance isolation for efficient multi-tenancy in RDMA.

Remote Procedure Call (RPC)

• HydraRPC: RPC in the CXL Era [Paper]

  • Alibaba & THU & ZJU & PKU

  • Utilize CXL-attached HDM to build RPC systems.

Journaling File System

• FastCommit: Resource-Efficient, Performant and Cost-Effective File System Journaling

  • Google

  • Best Paper Award

Rust-for-Linux

• An Empirical Study of Rust-for-Linux: The Success, Dissatisfaction, and Compromise [Paper] [Code]

  • BUPT & UESTC