unified memory

Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

2 months ago

Unified memory mini PCs like AMD Strix Halo and Apple M-series offer large memory capacity (up to 128GB or more) at relatively low cost, enabling them to run large models (e.g., 70B parameter models) that cannot fit into high-bandwidth discrete GPUs with limited VRAM (e.g., 32GB RTX 5090).
Performance for local LLMs is determined by two key specs: memory capacity (which dictates if a model loads) and memory bandwidth (which dictates text generation speed). Mini PCs excel in capacity but have lower bandwidth (typically 120-273 GB/s) compared to discrete GPUs (over 900 GB/s), resulting in slower decode speeds (e.g., 4-6 tokens/sec for dense 70B models).
Text generation (decode) is memory-bandwidth bound, meaning speed is limited by how fast data can be read from memory, not compute power. Prompt processing (prefill) is compute-bound and can be slow on mini PCs due to weaker integrated GPUs, causing long delays for long inputs (e.g., 40 seconds for a 4,000-token prompt).
Mixture-of-Experts (MoE) models are an exception, activating only a subset of parameters per token, allowing much faster decode speeds (e.g., 72 tokens/sec on Strix Halo) and making mini PCs feel faster despite limited bandwidth.
NPUs in these chips are largely irrelevant for local LLM chat due to memory bandwidth bottlenecks and lack of software support (e.g., in llama.cpp), with their primary use being small always-on tasks rather than running chatbots.
Owner reports confirm mini PCs can run models that won't fit on GPUs (e.g., Llama 3.3 70B on 128GB Strix Halo) but with slow decode speeds (single digits for dense 70B models) and software challenges, requiring patience for setup and updates.
Choosing a mini PC depends on use case: for small models (<8B), any 32GB unified box works; for 30B MoE models, 64-128GB boxes are ideal; for dense 70B models on a budget, 128GB Strix Halo or Macs offer capacity but slow speed; for fast performance or long prompts, discrete GPUs or cloud solutions are better.

Hasty Briefsbeta

unified memory

Apple Is the King of AI and Nobody Knows It

Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't