- Pretraining runs often fail due to breaking causality (e.g., expert choice routing where token allocation depends on future tokens) and adding bias (e.g., FP16 precision issues in collectives causing large accumulated errors).
- Breaking causality is problematic because it introduces information during training that is unavailable at inference, leading to model degradation; examples include Llama 4 (expert routing) and Gemini 2 Pro (token dropping).
- Bias is more dangerous than variance because it compounds, as seen in an early GPT-4 training bug where FP16 rounding caused a 10x error in gradient accumulation.
- There is debate on whether training failures are limited to a few known issues or whether new problems will emerge at each scale, with some experts believing bespoke issues will keep appearing.
- Kernel optimization for hardware like Blackwell is extremely difficult and may not be easily automated by AI, given Nvidia's struggles despite having top engineers.
- Inference for RL training is not the same as user-facing inference; numerical drift between training and inference engines can cause off-policy biases that matter for high-quality training.
- FSDP (fully sharded data parallel) is the default parallelism strategy due to easy overlap of compute and communication, but it fails when the number of GPUs scales beyond a crossover point where compute time drops below comms time.
- Pipeline parallelism introduces bubbles and architectural constraints (e.g., preventing efficient attention-to-residuals), slowing research iteration.
- Hierarchical collectives (reduce-scatter within a domain, then all-reduce across domains) optimize comms time and shift the crossover point for FSDP.
- Batch size floor limits pure FSDP: each GPU needs at least one sequence, so with a fixed sequence length, the number of GPUs is bounded by the number of sequences.