2 days ago
- PyTorch serves as both a reference language (clear, simplified) and an implementation language (production-ready).
- Reference implementations are used to verify correctness of optimized production implementations, especially when using kernel DSLs.
- Kernel DSLs allow explicit specification of tiling and data movement for peak performance, often maintained alongside a reference PyTorch implementation.
- Coding agents (LLMs) can generate explicit forward-backward code from autograd-based reference implementations, enabling separate optimization.
- A verifier (e.g., bitwise or structural equivalence) is needed to ensure the reference and optimized implementations remain equivalent.
- This approach combines the control of eager execution with the abstraction of graph-level optimization, addressing challenges in frontier training.