GPU Sharing

I am actively maintaining this list.

GPU Temporal Sharing

• Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications (EuroSys 2024) [Personal Notes] [Paper]

  • ETH

  • Intercept GPU kernel launches and schedule individual GPU operators

  • Utilize CUDA stream priorities; consider the PCIe bandwidth

  • Use NVIDIA Nsight Compute and NVIDIA Nsight Systems to collect the compute throughput, memory throughput, and execution time of each kernel

• Interference-aware Multiplexing for Deep Learning in GPU Clusters: A Middleware Approach (SC 2023) [Personal Notes] [Paper] [Code]

  • UMacau & SIAT, CAS

  • IADeep — a cluster scheduler on top of Kubernetes

  • Tune training configurations (e.g., batch size) across all co-located tasks; choose appropriate tasks to multiplex on a GPU device; consider PCIe bandwidth

• Transparent GPU Sharing in Container Clouds for Deep Learning Workloads (NSDI 2023) [Paper] [Code]

  • PKU

  • TGS: Transparent GPU sharing; adaptive rate control; unified memory.

• Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences (OSDI 2022) [Personal Notes] [Paper] [Code] [Benchmark] [Artifact]

  • SJTU

  • REEF: GPU kernel preemption; dynamic kernel padding.

• Gemini: Enabling Multi-Tenant GPU Sharing Based on Kernel Burst Estimation (TCC 2021) [Paper] [Code]

  • National Tsing Hua University

  • Enable fine-grained GPU allocation; launch kernels together.

• AntMan: Dynamic Scaling on GPU Clusters for Deep Learning (OSDI 2020) [Paper] [Code]

  • Alibaba

  • Enable GPU sharing in DL frameworks (TensorFlow/PyTorch); schedule operators.

• KubeShare: A Framework to Manage GPUs as First-Class and Shared Resources in Container Cloud (HPDC 2020) [Personal Notes] [Paper] [Code]

  • National Tsing Hua University

  • Kubernetes; CUDA API remoting.

• GaiaGPU: Sharing GPUs in Container Clouds (ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018) [Personal Notes] [Paper] [Code]

  • PKU & Tencent

  • Kubernetes; CUDA API remoting.

GPU Spatial Sharing

• Paella: Low-latency Model Serving with Software-defined GPU Scheduling (SOSP 2023) [Paper] [Code]

  • UPenn & DBOS, Inc.

  • Instrument kernels to expose at runtime, detailed information about the occupancy and utilization of the GPU's SMs

  • Compiler-library-scheduler co-design

• MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters (arXiv 2303.13803) [Paper]

  • PKU & ByteDance

  • Utilize NVIDIA MPS

• NVIDIA Multi-Instance GPU (MIG) [Homepage]

  • Partition the GPU into as many as seven instances, each fully isolated with its own high-bandwidth memory, cache, and compute cores.

  • Available for NVIDIA H100, A100, and A30 GPUs.

• NVIDIA Multi-Process Service (MPS) [Docs]

  • Transparently enable co-operative multi-process CUDA applications.

  • Terminating an MPS client without synchronizing with all outstanding GPU work (via Ctrl-C / program exception such as segfault / signals, etc.) can leave the MPS server and other MPS clients in an undefined state, which may result in hangs, unexpected failures, or corruptions.

• NVIDIA CUDA Multi-Stream [Docs]

  • Stream: a sequence of operations that execute in issue-order on the GPU.

  • Perform multiple CUDA operations simultaneously.

Survey

• GPU Virtualization and Scheduling Methods: A Comprehensive Survey (CSUR 2017) [Personal Notes] [Paper]

  • Queen’s University Belfast