21 days ago
- Matrix multiplication (matmul) is a core operation for LLMs, with over 83% of runtime in models like Llama 8B spent on matmul variants.
- GPUs offer massive parallelism (e.g., B200 handles up to 151,552 threads) and dedicated tensor cores for efficient matmul via MMA operations.
- Blackwell GPUs introduce 5th-gen tensor cores (tcgen05) supporting 256x256x16 MMA, tensor memory for reduced register usage, and enhanced pipelining.
- GPU programming uses a SIMT model where threads compute elements in parallel, organized into warps, blocks (CTAs), and clusters for memory sharing.
- Evolution from Ampere (async copies) to Hopper (TMA, WGMMA) to Blackwell (tensor memory) improves performance via overlapping data transfer and computation.
- A simple Mojo kernel for matmul on GPU achieves 5 TFLOPs, but cuBLAS reaches 1763 TFLOPs on B200, highlighting optimization potential using Blackwell features.