GPUs could explode to multiple TB with new storage-inspired memory tech

4 hours ago

HBF (high-bandwidth flash) stacks NAND flash layers to deliver SSD-like capacities (up to 512 GB per module) with HBM-like bandwidth (1.6 TB/s initially, targeting 3.2 TB/s eventually).
HBF is designed to supplement HBM for AI inference, not replace it, because NAND has limited write endurance and microsecond latency vs. nanosecond for DRAM.
In inference, HBF excels at the read-heavy decode phase (write-once, read-many), while HBM handles write-intensive prefill, making model weights readily available without reloading.
Larger models like multi-trillion-parameter MoE could fit on a single accelerator using HBF, reducing chip-to-chip interconnect bottlenecks.
Key challenges include standardization (under Open Compute Project), manufacturing complexities, and need for GPU/ASIC maker adoption; first samples expected late 2024 with devices early 2025.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Neutrino-1 models use a proprietary ternary-family weight format, delivering three models from one engine across datacenter GPUs, Apple Silicon, and desktop CPUs.
The flagship Neutrino-1 8B has 8.19 billion parameters stored in a single 3.88 GB file, with weights eight times smaller than fp16, fitting on 8 GB GPUs or 16 GB laptops.
Architecture: dense decoder-only transformer with grouped-query attention, requiring 144 KiB per token of KV cache.

Memory is crucial in AI infrastructure, increasing demand for high-performance DRAM and storage such as HBM, LPDDR, and fast SSDs.
AI has elevated memory from a commodity to a key strategic component, with major companies competing for advanced memory subsystems.
The current demand surge is driven by long-term expansion in use cases across servers, smartphones, automotive, and cloud, accelerated by AI.

<antirez>6 days ago

The author took a break from HNSW development to write a blog post sharing advanced findings, especially on making HNSWs fast enough for a low-latency, high-performance system like Redis.
The original HNSW paper lacks key details like deletion support and the necessity of multiple layers; the author modified HNSW to support actual removal of entries and questions if the 'H' (hierarchical) aspect is always beneficial.
Memory scaling is addressed by using 8-bit quantization per vector (computing max absolute value) which gives ~4x speedup, 4x reduction in vector size, and almost no recall loss; full precision and binary quantization options are also available.

Apple Neural Engine (ANE) is an efficient computing unit in Apple devices, introduced with A11 in 2017, offering ~19 TFLOPS FP16 performance in A19 Pro (2026).
ANE is optimized for convolution, not matrix multiplication (matmul), leading to hardware inefficiencies (~1/3 utilization) for LLMs like Gemma 4, requiring workarounds.
Gemma 4 E2B (2B params, INT4 quantized) and E4B (4B params) were split into 4 CoreML chunks to fit under 1 GB memory on iPhone 17 Pro (8 GB RAM), enabling ANE-only inference.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Neutrino-1 models use a proprietary ternary-family weight format, delivering three models from one engine across datacenter GPUs, Apple Silicon, and desktop CPUs.
The flagship Neutrino-1 8B has 8.19 billion parameters stored in a single 3.88 GB file, with weights eight times smaller than fp16, fitting on 8 GB GPUs or 16 GB laptops.
Architecture: dense decoder-only transformer with grouped-query attention, requiring 144 KiB per token of KV cache.

Memory is crucial in AI infrastructure, increasing demand for high-performance DRAM and storage such as HBM, LPDDR, and fast SSDs.
AI has elevated memory from a commodity to a key strategic component, with major companies competing for advanced memory subsystems.
The current demand surge is driven by long-term expansion in use cases across servers, smartphones, automotive, and cloud, accelerated by AI.

<antirez>6 days ago

The author took a break from HNSW development to write a blog post sharing advanced findings, especially on making HNSWs fast enough for a low-latency, high-performance system like Redis.
The original HNSW paper lacks key details like deletion support and the necessity of multiple layers; the author modified HNSW to support actual removal of entries and questions if the 'H' (hierarchical) aspect is always beneficial.
Memory scaling is addressed by using 8-bit quantization per vector (computing max absolute value) which gives ~4x speedup, 4x reduction in vector size, and almost no recall loss; full precision and binary quantization options are also available.

Apple Neural Engine (ANE) is an efficient computing unit in Apple devices, introduced with A11 in 2017, offering ~19 TFLOPS FP16 performance in A19 Pro (2026).
ANE is optimized for convolution, not matrix multiplication (matmul), leading to hardware inefficiencies (~1/3 utilization) for LLMs like Gemma 4, requiring workarounds.
Gemma 4 E2B (2B params, INT4 quantized) and E4B (4B params) were split into 4 CoreML chunks to fit under 1 GB memory on iPhone 17 Pro (8 GB RAM), enabling ANE-only inference.

Hasty Briefsbeta