- LLMs generate responses autoregressively during inference, building output one token at a time via sequential forward passes.
- Inference splits into a prefill phase (parallel processing of the prompt to fill the KV cache) and a decode phase (sequential generation of new tokens).
- The KV cache stores computed Key and Value vectors to avoid recomputation, saving significant work but consuming substantial memory, especially with long sequences.
- Decoding strategies like greedy decoding, top-k sampling, top-p (nucleus) sampling, and temperature scaling control token selection to balance predictability and creativity.
- Memory usage for the KV cache depends on model architecture, sequence length, and batch size, with techniques like Grouped Query Attention (GQA) reducing cache size.
- Beam search explores multiple token sequences for optimal outputs, while repetition penalty mitigates repetitive text in longer generations.
- The overall process involves tokenization, prefill, decode with caching, and detokenization to convert token IDs back to text.