Kimi K3 Inference self hosting study

7 hours ago

The author spent $300 renting GPUs to test whether reselling Kimi K3 tokens was profitable, finding the cost to serve is $12–16 per million output tokens, nearly equal to the $15 market price.
All major providers charge exactly $15 per million output tokens for K3, so price is commoditized; the real competitive fields are output speed and time-to-first-token.
A single 8×B300 box can generate roughly 3.3 billion output tokens per month at 90% utilization, yielding about $50k revenue but only ~$8k gross after $41k GPU cost; break-even requires ~75% utilization.
The first-token latency of 3–6 seconds is caused by CPU-side tokenization and API overhead, not the GPU, which accounts for only 2–6% of the wait.
Two potential optimizations—a faster tokenizer and DSpark speculative decoding—could lower costs to $6–8 per million and TTFT to ~1.5 seconds, making the $15 price viable.
Reselling tokens from self-hosted open models is a trap unless you have a specialist serving stack; self-hosting only makes sense for control, privacy, or high-utilization internal traffic.

AI pricing is based on tokens (roughly 3/4 of a word), with separate charges for input and output; output is typically 3-5x more expensive than input.
Prompt caching can reduce repeated input costs by up to 90%, and batch processing offers ~50% discounts, which many cost estimates ignore.
A live table shows pricing for 232+ flagship models, including GPT-5.x, Claude 5 variants, DeepSeek V4, Gemini 3.x, Grok 4.x, and GLM 5.2, with costs ranging from $0.09 to $10.00 per million input tokens.

Kimi K3 achieves 86.4% resolution rate but requires 8×B300 nodes and is slower than Claude Code.
Token costs for AI coding agents are rising sharply, with 90th percentile users spending ~$7,300/year.
Over 70% of API revenue from major model providers comes from coding use cases.

The author argues that AI inference is not as costly as commonly claimed, using H100 GPU pricing and DeepSeek R1 architecture as a baseline.
Input processing is extremely cheap (~$0.003 per million tokens) compared to output generation (~$3 per million tokens) due to sequential decode phase.
This cost asymmetry makes heavy-input, low-output use cases (e.g., coding assistants) highly profitable, with consumer plans showing 5-6x markup and developer plans up to 20x.

Comparing AI model costs by price per 1M tokens is meaningless due to differences in tokenizers and token efficiency.
Each frontier lab uses its own tokenizer, causing the same text to be split into varying token counts, making direct price comparisons unreliable.
Token efficiency—how much is achieved per token—varies widely, especially with 'thinking' tokens in chain-of-thought processes, significantly impacting overall costs.

The argument that self-hosting LLMs saves money is inaccurate when considering utility and hardware costs, unless prioritizing data security.
Local inference on high-end GPUs like an RTX 4090 consumes 0.025 kWh for 18k tokens, costing ~$0.0095 in electricity, which is 2.5x to 3.5x higher than centralized API costs such as DeepSeek V4 Flash at ~$0.0038.
Idle power consumption for local systems adds significant costs: a dual-GPU Nvidia desktop idling at 90W costs $898 over three years, making total ownership 26x higher than API costs for 1M tokens/day.

AI pricing is based on tokens (roughly 3/4 of a word), with separate charges for input and output; output is typically 3-5x more expensive than input.
Prompt caching can reduce repeated input costs by up to 90%, and batch processing offers ~50% discounts, which many cost estimates ignore.
A live table shows pricing for 232+ flagship models, including GPT-5.x, Claude 5 variants, DeepSeek V4, Gemini 3.x, Grok 4.x, and GLM 5.2, with costs ranging from $0.09 to $10.00 per million input tokens.

Kimi K3 achieves 86.4% resolution rate but requires 8×B300 nodes and is slower than Claude Code.
Token costs for AI coding agents are rising sharply, with 90th percentile users spending ~$7,300/year.
Over 70% of API revenue from major model providers comes from coding use cases.

The author argues that AI inference is not as costly as commonly claimed, using H100 GPU pricing and DeepSeek R1 architecture as a baseline.
Input processing is extremely cheap (~$0.003 per million tokens) compared to output generation (~$3 per million tokens) due to sequential decode phase.
This cost asymmetry makes heavy-input, low-output use cases (e.g., coding assistants) highly profitable, with consumer plans showing 5-6x markup and developer plans up to 20x.

Comparing AI model costs by price per 1M tokens is meaningless due to differences in tokenizers and token efficiency.
Each frontier lab uses its own tokenizer, causing the same text to be split into varying token counts, making direct price comparisons unreliable.
Token efficiency—how much is achieved per token—varies widely, especially with 'thinking' tokens in chain-of-thought processes, significantly impacting overall costs.

The argument that self-hosting LLMs saves money is inaccurate when considering utility and hardware costs, unless prioritizing data security.
Local inference on high-end GPUs like an RTX 4090 consumes 0.025 kWh for 18k tokens, costing ~$0.0095 in electricity, which is 2.5x to 3.5x higher than centralized API costs such as DeepSeek V4 Flash at ~$0.0038.
Idle power consumption for local systems adds significant costs: a dual-GPU Nvidia desktop idling at 90W costs $898 over three years, making total ownership 26x higher than API costs for 1M tokens/day.

Hasty Briefsbeta