- LLMs generate responses autoregressively during inference, building output one token at a time via sequential forward passes.
- Inference splits into a prefill phase (parallel processing of the prompt to fill the KV cache) and a decode phase (sequential generation of new tokens).
- The KV cache stores computed Key and Value vectors to avoid recomputation, saving significant work but consuming substantial memory, especially with long sequences.
- Decoding strategies like greedy decoding, top-k sampling, top-p (nucleus) sampling, and temperature scaling control token selection to balance predictability and creativity.