OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training
17 hours ago
- #LLM Training
- #Memory Optimization
- #Pipeline Parallelism
- OptPipe introduces a memory- and scheduling-optimized pipeline parallelism approach for LLM training.
- Existing pipeline parallelism techniques are heuristic and coarse-grained, lacking fine-grained optimization.
- The paper formulates scheduling as a constrained optimization problem considering memory capacity, activation reuse, and pipeline bubble minimization.
- The proposed method dynamically optimizes trade-offs between memory and time based on model structure and hardware configuration.
- Experimental results show up to 50% reduction in idle pipeline time and improved throughput and memory utilization.
- The approach enables training larger models within limited memory budgets.