How many GPUs is 1M/B/T tokens?
a day ago
- The calculator estimates GPU count needed for a given token volume and model, with a confidence range (e.g., 1 trillion tokens/month on Llama 3.3 70B needs ~367 H100 GPUs).
- Calculation steps: convert monthly tokens to per-second, adjust for token mix (input/output ratio) and utilization, then divide by GPU throughput.
- Assumptions like GPU speed, input cost, memory, and model type affect results; the most sensitive assumption is highlighted for each model.
- GPU performance tables list low/base/high tokens per second for various models (e.g., Llama 3.3 70B, DeepSeek, GLM) on A100, V100, H100, B200, with sources.
- Results are planning estimates with expected 2x error; real hardware measurement is recommended before purchase.
- The calculator does not include prompt caching, speculative decoding, reasoning overhead, failures, or regional duplication.