Bilingual

What happens when you run a CUDA kernel?
25 days ago
- CUDA programs are compiled by nvcc into host code and device code, with the device code undergoing transformations from PTX to SASS.
- The host code uses a stub to pack kernel arguments and triggers the GPU via the CUDA runtime and driver, involving ioctls and a doorbell register.
- The GPU executes kernels via a work distributor assigning blocks to SMs, with warps scheduled using compiler-encoded control bits to manage dependencies and stalls.
- Memory accesses are coalesced, leveraging caches and DRAM, with performance often limited by memory bandwidth for low arithmetic intensity kernels.
- Completion is signaled via semaphores, allowing asynchronous execution and data transfer back to the host for output.
Reverse-engineering Nvidia's CUDA-checkpoint for faster cold starts
15 days ago
- NVIDIA's cuda-checkpoint allows freezing a CUDA process and serializing its GPU state into host memory, later restoring it exactly as it was.
- Checkpointing involves transferring GPU state into a host-side anonymous buffer, which can be manipulated directly (e.g., modifying a GPU counter) before restoration.
- The checkpoint operation is performed by a service thread within the target process, communicating via pipes, and uses ordinary driver ioctls rather than a dedicated checkpoint syscall.
- Checkpoint and restore speeds are limited by the allocation and deallocation of staging buffers, with checkpoint copying at ~3 GiB/s and restore at ~8 GiB/s, well below PCIe bandwidth limits.
- Performance can be improved by using Transparent Huge Pages (THP) to reduce buffer zeroing costs, or by intercepting mmap calls to reuse pre-zeroed buffers, achieving up to 4x speedup.
AMD calls CUDA a 'non-event'
4 hours ago
- Nvidia's dominance in AI hardware is largely due to its CUDA software platform, but AMD claims CUDA is becoming irrelevant for many companies.
- AMD's executive says customers are now programming at higher levels of abstraction, bypassing CUDA entirely, and AI agents are helping optimize platforms without it.
- If CUDA's moat erodes, cheaper alternatives like AMD's ROCm could lure AI companies away from Nvidia hardware, threatening Nvidia's market lead.
- However, skepticism remains because Nvidia's established ecosystem and proven infrastructure may still be favored, especially for high-stakes AI data centers.

Hasty Briefsbeta

#cuda

What happens when you run a CUDA kernel?

Reverse-engineering Nvidia's CUDA-checkpoint for faster cold starts

AMD calls CUDA a 'non-event'