SC 2024

Meta Info

Homepage: https://sc24.conference-program.com

Paper list: https://dl.acm.org/doi/proceedings/10.5555/3703596

Papers

AI Infrastructure

• Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning [Paper] [HAI Platform Code]

  • DeepSeek AI

  • Include Network Co-Design, HFReduce (collective communication library), HaiScale (optimized parallelism methods), 3FS Distributed File System, and HAI Platform (task scheduling, fault tolerance).

Large Language Models (LLMs)

• LLM inference

  • PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation [Paper] [Code]

    • Iowa State University & TU Darmstadt

    • Continuous Asynchronous Speculation: run single-token inference simultaneously with several speculative runs.

    • Early Inference Cancellation: skip the computation of invalidated runs.

  • LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services [Paper] [Benchmark] [Code]

    • IBM Research

    • Learn a predictive model to recommend the most cost-effective hardware for a previously unseen LLM.

• LLM fine-tuning

  • Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity [Paper] [Code]

    • MSRA & THU

• LLM for anomaly detection

  • Large Language Models for Anomaly Detection in Computational Workflows: From Supervised Fine-Tuning to In-Context Learning [Paper] [Code] [Benchmark]

    • Argonne National Laboratory & USC & Oak Ridge National Laboratory

    • Investigated two approaches: (1) supervised fine-tuning (pre-trained LLMs are fine-tuned on labeled data for sentence classification to identify anomalies); (2) in-context learning (prompts containing task descriptions and examples guide LLMs in few-shot anomaly detection without fine-tuning).

Mixture-of-Experts (MoEs)

• APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes [Paper] [Code]

  • SYSU

Deep Learning Recommendation Models (DLRMs)

• Accelerating Distributed DLRM Training with Optimized TT Decomposition and Micro-Batching [Paper] [Code]

  • WHU & NVIDIA & UMacau

  • EcoRec: eliminate redundancy in TT (Tensor-Train) operations; micro-batching with sorted indices to reduce memory.

• Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression [Paper] [Code]

  • Indiana University, Bloomington & Meta & University of Rochester & ICT, CAS

  • In-depth analysis of embedding data features; employ error-bounded lossy compression to reduce the communication data size.

• Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link [Paper] [Code]

  • UC Merced & SK Hynix

  • TECO: Tensor-CXL-Offload

  • Introduce a cache coherence interconnect based on CXL to build a cache coherence domain between CPU memory and accelerator memory; offload tensors to CPU memory to save accelerator memory.

• RecFlex: Enabling Feature Heterogeneity-Aware Optimization for Deep Recommendation Models with Flexible Schedules [Paper] [Code]

  • RUC & Microsoft & UCSD

  • Create fused kernels with distinct schedules for different feature fields.

Graph Transformer

• TorchGT: A Holistic System for Large-Scale Graph Transformer Training [Paper] [Code]

  • NTU & Shanghai AI Lab & ZJU & SenseTime

Reinforcement Learning (RL)

• Stellaris: Staleness-Aware Distributed Reinforcement Learning with Serverless Computing [Paper] [Code]

  • Stevens Institute of Technology & NEU & Stony Brook University & Missouri University of Science and Technology

  • Introduce a generic asynchronous learning paradigm.

Job Scheduling

• PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters [Paper] [Code]

  • UW-Madison

  • Characterize which applications are more likely to suffer from performance variability; balance performance variability with locality to ensure jobs are spread across as few nodes as possible.

Distributed Training

• ,Optimizing Distributed ML Communication with Fused Computation-Collective Operations [Paper]

  • AMD

  • Developed three prototype fused operators (embedding + All-to-All, GEMV + AllReduce, and GEMM + All-to-All) to address the communication overheads in DLRM, Transformers and MoE model architectures.

Serverless Computing

• SMIless: Serving DAG-based Inference with Dynamic Invocations under Serverless Computing [Paper] [Code]

  • SIAT, CAS & UMacau

  • Integrate adaptive pre-warming windows; built on top of OpenFaaS.

GPU Sharing

• ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments [Paper] [Code]

  • Chung-Ang University & Electronics and Telecommunications Research Institute & Virginia Tech

  • Integrate MIG and MPS to enhance GPU utilization.

Performance Analysis

• GVARP: Detecting Performance Variance on Large-Scale Heterogeneous Systems [Paper] [Code]

  • Beihang University

  • Employ static analysis to identify the performance-critical parameters of kernel functions; segment the program execution with external library calls and asynchronous kernel operations; construct a state transfer graph and estimate the workload of each program segment.

Interconnects

• Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects [Paper] [Benchmark]

  • Sapienza University of Rome & University of Trento & Vrije Universiteit Amsterdam & ETH & CINECA & University of Antwerp & HPE & NVIDIA

  • Characterize three supercomputers: Alps, Leonardo, and LUMI.

Acronyms

• LLM: Large Language Model

• MoE: Mixture-of-Experts

• DLRM: Deep Learning Recommendation Model

• PEFT: Parameter-Efficient Fine-Tuning

• MIG: Multi-Instance GPU

• MPS: Multi-Process Service

• CXL: Compute Express Link