7 hours ago
- The theoretical best LLM output is deterministic by picking the highest probability token, but that is not ideal for many applications.
- Top-p (nucleus sampling) collects top-scoring tokens up to a probability sum 'p' and performs weighted random sampling, but it may include very weak tokens.
- The proposed First Token Cutoff (FTC) algorithm selects tokens only if their probability is within a certain ratio (co) of the top token's probability, bounding the worst possible token.
- FTC gives a single, linear, understandable parameter 'co', unlike top-p whose effect depends on the distribution shape.
- The avalanche effect (changing context) can provide variability without needing to pick weak tokens.
- The author advocates for more research into sampling algorithms and logits analysis to improve LLM output quality and uncertainty detection.