Bilingual

Popping the GPU Bubble
22 days ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.
AI inference is obviously profitable
18 days ago
- The belief that AI inference is unprofitable due to high costs is disputed, as evidence suggests it can be profitable.
- Rough cost estimates indicate inference costs around $1 per million tokens, with GPT-5.4-mini charging $4.50, implying high profit margins.
- Open LLMs like DeepSeek report over 80% profit margins on inference, with market costs near $0.87 per million tokens, supporting profitability.
- AI labs like OpenAI and Anthropic may use high inference margins to subsidize training costs, explaining their higher prices.
- Even if major AI labs fail, inference providers without training costs can profitably serve models, showing the inference business is sustainable.
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
18 days ago
- AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
- Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
- Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.
- AMD faces challenges with day-0 support and software friction for frontier models, but the performance gap is narrowing as kernel and model optimizations improve, reducing reliance on custom kernels.
- The study highlights that achieving state-of-the-art performance on AMD is increasingly about support and optimization rather than inherent software limitations, indicating erosion of NVIDIA's CUDA advantage.
Mesh LLM: distributed AI computing on iroh
10 days ago
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.
- Inside the main connection, various stream types (e.g., gossip, HTTP tunnels, route queries) are demuxed by a leading byte, enabling efficient peer-to-peer operations.
- Users can install lightweight software (~18 MB) to join the public mesh or set up private deployments, accessible via localhost:9337/v1 for standard OpenAI clients.
Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
6 days ago
- Author successfully runs Google's Gemma 4 26B model on a 13-year-old dual Xeon server with no GPU, achieving about 5 tokens per second decoding speed.
- The project required patching the ik_llama.cpp fork to work on pre-AVX2 CPUs, with Claude AI assisting in diagnosing and fixing silent kernel fall-through bugs in the model's graph ops.
- The fix involves compile-time adjustments and runtime workarounds to handle missing AVX2 instructions, enabling older hardware to run modern MoE models cost-effectively as a local fallback.
Show HN: FlexInference LLM Router
3 days ago
- FlexInference is a service that routes AI inference requests to find cheaper options within a specified time budget.
- If cheaper inference isn't found, the request escalates to standard tier at no extra cost; routing is free.
- Users bring their own provider keys and are billed directly, with FlexInference taking a 20% fee on savings.
- It supports OpenAI, Anthropic, Gemini, and integrates with platforms like Amazon Bedrock and Google Vertex AI.
- The service ensures data privacy by not storing prompts or replies and using envelope encryption for keys.
- It provides transparent error handling, strict parameter adherence, and works with SDKs or direct API calls.
- An MCP server allows coding tools to manage keys and search docs via OAuth, without handling raw keys.
- Founded by Adi to improve access to AI models by reducing costs and layers that slow down and gatekeep inference.
Google just bet its inference future on a chip built for one model
3 hours ago
- Google's Frozen v2 chip hardwires parts of Gemini's architecture but keeps weights updatable to avoid obsolescence.
- The chip aims to deliver 6-10 times more tokens per watt than current Google AI chips, helping alleviate the AI compute crunch.
- This represents a shift toward model-specific silicon, similar to the evolution of Bitcoin mining from GPUs to ASICs.
- Competitors like Taalas, d-Matrix, and SambaNova are also developing specialized hardware for AI inference.
- The approach balances hyper-efficiency of hardwired architectures with flexibility for future model updates.

Hasty Briefsbeta

#ai inference

Popping the GPU Bubble

AI inference is obviously profitable

GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell

Mesh LLM: distributed AI computing on iroh

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

Show HN: FlexInference LLM Router

Google just bet its inference future on a chip built for one model