Reverse-engineering Nvidia's CUDA-checkpoint for faster cold starts
6 hours ago
- #reverse-engineering
- #CUDA
- #GPU-checkpoint
- NVIDIA's cuda-checkpoint allows freezing a CUDA process and serializing its GPU state into host memory, later restoring it exactly as it was.
- Checkpointing involves transferring GPU state into a host-side anonymous buffer, which can be manipulated directly (e.g., modifying a GPU counter) before restoration.
- The checkpoint operation is performed by a service thread within the target process, communicating via pipes, and uses ordinary driver ioctls rather than a dedicated checkpoint syscall.
- Checkpoint and restore speeds are limited by the allocation and deallocation of staging buffers, with checkpoint copying at ~3 GiB/s and restore at ~8 GiB/s, well below PCIe bandwidth limits.
- Performance can be improved by using Transparent Huge Pages (THP) to reduce buffer zeroing costs, or by intercepting mmap calls to reuse pre-zeroed buffers, achieving up to 4x speedup.