5 hours ago
- Training large language models with billions of parameters requires distributed computing across hundreds or thousands of GPUs, where hardware failures are expected.
- Traditional fault-tolerance relies on periodic checkpointing, which has drawbacks like overhead, wasted computation, cluster idle time, and scalability limits.
- PyTorch Monarch introduces a new distributed programming paradigm with actor-based runtime, process mesh abstraction, and asynchronous execution for elastic, fault-tolerant training.
- Monarch was ported to AMD Instinct GPUs with ROCm by adapting collective communications, GPU memory management, and RDMA integration.
- The port used hipify_torch for C++ bridge code, auto-detection for platform-specific builds, and a Rust compatibility shim to avoid forking bindings.
- A case study demonstrated fault-tolerant training on AMD GPUs using Monarch, TorchTitan, and TorchFT, with dynamic recovery from failures without global restart.
- Performance tests on SLURM and Kubernetes clusters showed stable training with minimal disruption despite frequent injected failures.
- Monarch's actor-based runtime and supervision trees isolate failures, enabling rapid local recovery and quorum-based synchronization for healthy nodes.