Bilingual

Popping the GPU Bubble
a month ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.
AI inference is obviously profitable
22 days ago
- The belief that AI inference is unprofitable due to high costs is disputed, as evidence suggests it can be profitable.
- Rough cost estimates indicate inference costs around $1 per million tokens, with GPT-5.4-mini charging $4.50, implying high profit margins.
- Open LLMs like DeepSeek report over 80% profit margins on inference, with market costs near $0.87 per million tokens, supporting profitability.
- AI labs like OpenAI and Anthropic may use high inference margins to subsidize training costs, explaining their higher prices.
- Even if major AI labs fail, inference providers without training costs can profitably serve models, showing the inference business is sustainable.
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
22 days ago
- AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
- Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
- Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.
- AMD faces challenges with day-0 support and software friction for frontier models, but the performance gap is narrowing as kernel and model optimizations improve, reducing reliance on custom kernels.
- The study highlights that achieving state-of-the-art performance on AMD is increasingly about support and optimization rather than inherent software limitations, indicating erosion of NVIDIA's CUDA advantage.
Mesh LLM: distributed AI computing on iroh
14 days ago
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.
- Inside the main connection, various stream types (e.g., gossip, HTTP tunnels, route queries) are demuxed by a leading byte, enabling efficient peer-to-peer operations.
- Users can install lightweight software (~18 MB) to join the public mesh or set up private deployments, accessible via localhost:9337/v1 for standard OpenAI clients.
Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
10 days ago
- Author successfully runs Google's Gemma 4 26B model on a 13-year-old dual Xeon server with no GPU, achieving about 5 tokens per second decoding speed.
- The project required patching the ik_llama.cpp fork to work on pre-AVX2 CPUs, with Claude AI assisting in diagnosing and fixing silent kernel fall-through bugs in the model's graph ops.
- The fix involves compile-time adjustments and runtime workarounds to handle missing AVX2 instructions, enabling older hardware to run modern MoE models cost-effectively as a local fallback.
Show HN: FlexInference LLM Router
7 days ago
- FlexInference is a service that routes AI inference requests to find cheaper options within a specified time budget.
- If cheaper inference isn't found, the request escalates to standard tier at no extra cost; routing is free.
- Users bring their own provider keys and are billed directly, with FlexInference taking a 20% fee on savings.
- It supports OpenAI, Anthropic, Gemini, and integrates with platforms like Amazon Bedrock and Google Vertex AI.
- The service ensures data privacy by not storing prompts or replies and using envelope encryption for keys.
- It provides transparent error handling, strict parameter adherence, and works with SDKs or direct API calls.
- An MCP server allows coding tools to manage keys and search docs via OAuth, without handling raw keys.
- Founded by Adi to improve access to AI models by reducing costs and layers that slow down and gatekeep inference.
Google just bet its inference future on a chip built for one model
4 days ago
- Google's Frozen v2 chip hardwires parts of Gemini's architecture but keeps weights updatable to avoid obsolescence.
- The chip aims to deliver 6-10 times more tokens per watt than current Google AI chips, helping alleviate the AI compute crunch.
- This represents a shift toward model-specific silicon, similar to the evolution of Bitcoin mining from GPUs to ASICs.
- Competitors like Taalas, d-Matrix, and SambaNova are also developing specialized hardware for AI inference.
- The approach balances hyper-efficiency of hardwired architectures with flexibility for future model updates.
AMD and Cerebras Launch AI Inference Solution
a day ago
- AMD and Cerebras Systems announced a technical partnership to deliver a new disaggregated AI inference solution.
- The solution combines AMD Helios rackscale solutions with the Cerebras Wafer-Scale Engine for ultra-low-latency and high throughput.
- The joint solution is expected to deliver up to 5x higher tokens per second per watt (T/s/W) compared to a Cerebras WSE-only configuration.
- AMD Helios provides high-throughput prompt processing, while Cerebras WSE accelerates memory-bandwidth-intensive token generation with ultra-low latency.
- The solution addresses the growing diversity of AI inference workloads with heterogeneous infrastructure.
- Cerebras plans to deploy AMD Helios in its data centers, with availability through Cerebras Cloud in the second half of 2026.

Hasty Briefsbeta

ai inference

Popping the GPU Bubble

AI inference is obviously profitable

GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell

Mesh LLM: distributed AI computing on iroh

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

Show HN: FlexInference LLM Router

Google just bet its inference future on a chip built for one model

AMD and Cerebras Launch AI Inference Solution