a month ago
- Matrix multiplication (matmul) is a core operation for LLMs, with over 83% of runtime in models like Llama 8B spent on matmul variants.
- GPUs offer massive parallelism (e.g., B200 handles up to 151,552 threads) and dedicated tensor cores for efficient matmul via MMA operations.
- Blackwell GPUs introduce 5th-gen tensor cores (tcgen05) supporting 256x256x16 MMA, tensor memory for reduced register usage, and enhanced pipelining.
- GPU programming uses a SIMT model where threads compute elements in parallel, organized into warps, blocks (CTAs), and clusters for memory sharing.