- A bug in the evaluation code for GPT-2 style models was identified: `model.state_dict()` returned references to live parameters instead of copies, causing `model.load_state_dict()` to be a no-op during early stopping.
- The bug was fixed by using `deepcopy(model.state_dict())` to store parameter snapshots when validation loss decreased.
- Additionally, the evaluation code was updated to use all validation batches instead of only the first five, improving accuracy of early stopping.
- After fixing the bug, a new baseline was generated by re-running the instruction-following eval on all models, with results showing that OpenAI's GPT-2 weights still outperform the author's models.
- The ranking changes were minor overall, but some models (e.g., JAX without MHA bias) improved significantly, while others (e.g., 1xrtx3090-stacked-interventions) dropped, possibly due to earlier bug benefits.
- The author concludes that the bug fix ensures the eval works as intended and does not alter the core mystery of why OpenAI's models are better.
- Future experiments will focus on modifying training to close the performance gap with OpenAI's models.