Hasty Briefsbeta

OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training

17 hours ago
  • #LLM Training
  • #Memory Optimization
  • #Pipeline Parallelism
  • OptPipe introduces a memory- and scheduling-optimized pipeline parallelism approach for LLM training.
  • Existing pipeline parallelism techniques are heuristic and coarse-grained, lacking fine-grained optimization.
  • The paper formulates scheduling as a constrained optimization problem considering memory capacity, activation reuse, and pipeline bubble minimization.
  • The proposed method dynamically optimizes trade-offs between memory and time based on model structure and hardware configuration.
  • Experimental results show up to 50% reduction in idle pipeline time and improved throughput and memory utilization.
  • The approach enables training larger models within limited memory budgets.