ai inference

Bilingual

GPUs could explode to multiple TB with new storage-inspired memory tech
2 months ago
- HBF (high-bandwidth flash) stacks NAND flash layers to deliver SSD-like capacities (up to 512 GB per module) with HBM-like bandwidth (1.6 TB/s initially, targeting 3.2 TB/s eventually).
- HBF is designed to supplement HBM for AI inference, not replace it, because NAND has limited write endurance and microsecond latency vs. nanosecond for DRAM.
- In inference, HBF excels at the read-heavy decode phase (write-once, read-many), while HBM handles write-intensive prefill, making model weights readily available without reloading.
- Larger models like multi-trillion-parameter MoE could fit on a single accelerator using HBF, reducing chip-to-chip interconnect bottlenecks.
- Key challenges include standardization (under Open Compute Project), manufacturing complexities, and need for GPU/ASIC maker adoption; first samples expected late 2024 with devices early 2025.
Running Kimi K3 on MI355X at Better Performance per Dollar Than B300
2 months ago
- AMD MI355X achieves ~952 tok/s/node for Kimi K3, outperforming B200 on performance per dollar due to lower cost and higher HBM capacity.
- Kimi K3 is a 2.8T-parameter open-source model that cannot fit on a single B200 node, requiring two nodes or a B300 node with 288GB VRAM per GPU.
- AMD MI355X offers comparable specs to B300 at ~2.4× lower cost per GPU, making it a cost-efficient alternative despite software challenges.
- Software issues on ROCm were resolved with simple fixes (e.g., adding top-k renorm function for speculative decoding, padding head count for MLA prefill), not requiring custom kernels.
- Speculative decoding improved single-stream throughput by ~2.2× and peak aggregate by +18%, while prefill optimizations cut time-to-first-token by 2–3×.
- The MI355X's large HBM capacity gives it a practical edge over B200 for models like Kimi K3 that require >1.5TB VRAM for weights and KV cache.
AI inference is obviously profitable
2 months ago
- AI inference is profitable, with frontier providers reporting 70-80% gross margins.
- Cost estimates show inference costs around $1 per million tokens, while pricing is $4.50-$27 per million tokens, supporting high margins.
- Open-weight models like DeepSeek-R1 charge less than half of OpenAI/Anthropic, yet still maintain over 80% profit margins.
- DeepSeek-V4-Pro inference costs on the market are about $0.87 per million tokens, close to the actual service cost.
- AI labs like OpenAI and Anthropic use high inference margins to subsidize training costs, keeping prices high for profit.
- Inference-only providers don't need to fund training, so inference remains profitable even if AI labs fail.
AMD and Cerebras Launch AI Inference Solution
2 months ago
- AMD and Cerebras Systems announced a technical partnership to deliver a new disaggregated AI inference solution.
- The solution combines AMD Helios rackscale solutions with the Cerebras Wafer-Scale Engine for ultra-low-latency and high throughput.
- The joint solution is expected to deliver up to 5x higher tokens per second per watt (T/s/W) compared to a Cerebras WSE-only configuration.
- AMD Helios provides high-throughput prompt processing, while Cerebras WSE accelerates memory-bandwidth-intensive token generation with ultra-low latency.
- The solution addresses the growing diversity of AI inference workloads with heterogeneous infrastructure.
- Cerebras plans to deploy AMD Helios in its data centers, with availability through Cerebras Cloud in the second half of 2026.
Google just bet its inference future on a chip built for one model
2 months ago
- Google's Frozen v2 chip hardwires parts of Gemini's architecture but keeps weights updatable to avoid obsolescence.
- The chip aims to deliver 6-10 times more tokens per watt than current Google AI chips, helping alleviate the AI compute crunch.
- This represents a shift toward model-specific silicon, similar to the evolution of Bitcoin mining from GPUs to ASICs.
- Competitors like Taalas, d-Matrix, and SambaNova are also developing specialized hardware for AI inference.
- The approach balances hyper-efficiency of hardwired architectures with flexibility for future model updates.
Show HN: FlexInference LLM Router
2 months ago
- FlexInference is a service that routes AI inference requests to find cheaper options within a specified time budget.
- If cheaper inference isn't found, the request escalates to standard tier at no extra cost; routing is free.
- Users bring their own provider keys and are billed directly, with FlexInference taking a 20% fee on savings.
- It supports OpenAI, Anthropic, Gemini, and integrates with platforms like Amazon Bedrock and Google Vertex AI.
- The service ensures data privacy by not storing prompts or replies and using envelope encryption for keys.
- It provides transparent error handling, strict parameter adherence, and works with SDKs or direct API calls.
- An MCP server allows coding tools to manage keys and search docs via OAuth, without handling raw keys.
- Founded by Adi to improve access to AI models by reducing costs and layers that slow down and gatekeep inference.
Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
2 months ago
- Author successfully runs Google's Gemma 4 26B model on a 13-year-old dual Xeon server with no GPU, achieving about 5 tokens per second decoding speed.
- The project required patching the ik_llama.cpp fork to work on pre-AVX2 CPUs, with Claude AI assisting in diagnosing and fixing silent kernel fall-through bugs in the model's graph ops.
- The fix involves compile-time adjustments and runtime workarounds to handle missing AVX2 instructions, enabling older hardware to run modern MoE models cost-effectively as a local fallback.
Mesh LLM: distributed AI computing on iroh
2 months ago
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.
- Inside the main connection, various stream types (e.g., gossip, HTTP tunnels, route queries) are demuxed by a leading byte, enabling efficient peer-to-peer operations.
- Users can install lightweight software (~18 MB) to join the public mesh or set up private deployments, accessible via localhost:9337/v1 for standard OpenAI clients.
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
3 months ago
- AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
- Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
- Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.
- AMD faces challenges with day-0 support and software friction for frontier models, but the performance gap is narrowing as kernel and model optimizations improve, reducing reliance on custom kernels.
- The study highlights that achieving state-of-the-art performance on AMD is increasingly about support and optimization rather than inherent software limitations, indicating erosion of NVIDIA's CUDA advantage.
AI inference is obviously profitable
3 months ago
- The belief that AI inference is unprofitable due to high costs is disputed, as evidence suggests it can be profitable.
- Rough cost estimates indicate inference costs around $1 per million tokens, with GPT-5.4-mini charging $4.50, implying high profit margins.
- Open LLMs like DeepSeek report over 80% profit margins on inference, with market costs near $0.87 per million tokens, supporting profitability.
- AI labs like OpenAI and Anthropic may use high inference margins to subsidize training costs, explaining their higher prices.
- Even if major AI labs fail, inference providers without training costs can profitably serve models, showing the inference business is sustainable.
Popping the GPU Bubble
3 months ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.

ai inference

GPUs could explode to multiple TB with new storage-inspired memory tech

Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

AI inference is obviously profitable

AMD and Cerebras Launch AI Inference Solution

Google just bet its inference future on a chip built for one model

Show HN: FlexInference LLM Router

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

Mesh LLM: distributed AI computing on iroh

GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell

AI inference is obviously profitable

Popping the GPU Bubble