> For the complete documentation index, see [llms.txt](https://paper.lingyunyang.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://paper.lingyunyang.com/reading-notes/conference/sigcomm-2022/multi-resource-interleaving-for-deep-learning-training.md).

# Multi-resource interleaving for deep learning training

## Meta Info

Presented in [SIGCOMM 2022](https://doi.org/10.1145/3544216.3544224).

Authors: Yihao Zhao, Yuanqiang Liu, Yanghua Peng, Yibo Zhu, Xuanzhe Liu, Xin Jin.

Code: <https://github.com/Rivendile/Muri>

## Understanding the paper

### TL;DRs

This paper presents **Muri**, a DL *cluster scheduler* that utilizes *multi-resource interleaving* to improve cluster and job efficiency.\
It designs a scheduling algorithm based on *Blossom algorithm* for multi-resource multi-job packing.

### Key Insights

* DL training jobs have a unique staged, *iterative computation pattern*.
  * Motivate fine-grained multi-resource interleaving in *time*.
  * Each iteration is composed of a sequence of stages like *data loading (Storage IO)*, *preprocessing (CPUs)*, *forward and backward propagation (GPUs)*, and *gradient synchronization (Network IO)*.

### Implementation

Built a prototype of Muri with \~7k LoC and integrated it with **PyTorch**.