apple silicon

Bilingual

Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)
2 months ago
- Benchmarks for LLM inference on Apple Silicon were measured using a production fleet with Ollama API, fixed prompt, and 512-token generation budget.
- Results for M4 Mac mini (16 GB, 120 GB/s) include models from 3B to 14B parameters, with generation speeds ranging from 46.7 tok/s (Llama 3.2 3B) to 11.7 tok/s (Qwen 2.5 14B).
- Future benchmarks will cover M4 Pro, M4 Max, and M3 Ultra chips, with performance expected to scale with memory bandwidth.
- Methodology: Ollama 0.31.2 on macOS 15.3.1, default Q4_K_M quantization, 3 recorded runs after warm-up, median speeds reported. Raw data available under CC BY 4.0.
Distributing LLM Inference in DwarfStar
2 months ago
- High-end NVIDIA setups require expensive servers and power; alternatives like Apple hardware and DGX Spark exist but have limitations.
- Mac Studio M3 Ultra 512GB can run DeepSeek v4 PRO at usable speeds (150 t/s prefill, 10-13 t/s decoding) for around $12k.
- The future is uncertain: NVIDIA prices are unlikely to drop, and RAM shortages may delay higher-end Apple hardware like M5 Ultra.
- Currently, MacBook M5 Max 128GB offers the best value, running 2-bit quantized models at ~500 t/s prefill and ~35-40 t/s decoding for $6-7k.
- Distributed inference is promising: traditional layer splitting and expert parallelism across machines, but communication speed is a bottleneck.
- A novel third approach is model ensemble, where different models run independently and combine logits or select best continuations, showing improved performance.
- The author expresses interest in further exploring these distributed techniques for local inference.
M-Chips: M7 with up to 1.5 TB – and why Apple is skipping the M6
2 months ago
- Apple is skipping M6 Pro, Max, and Ultra variants, planning only a standard M6 chip.
- The M7 series, starting in 2027, will feature major AI accelerator upgrades and include Pro, Max, and Ultra variants with up to 1.5 TB addressable RAM.
- Apple emphasizes local AI for cost savings, privacy, and technical advancements, aligning with trends away from cloud-dependent inference.
Mac gaming is finally getting the overpowered upgrade it deserves
2 months ago
- Game Porting Toolkit 4 beta significantly boosts Mac gaming performance, with GTA V achieving a 66% FPS increase on M4 Pro MacBook Pro.
- The toolkit translates Windows DirectX to Apple's Metal API in real-time, enabling play of Windows-exclusive games and aiding developers in evaluating Mac ports.
- Performance improvements stem from reduced translation overhead, allowing Apple Silicon to focus more on rendering rather than instruction conversion.
- While not all games see dramatic gains, the software advances suggest Mac hardware is no longer the bottleneck, positioning Apple Silicon as a viable gaming platform.
- GPTK serves both developers for port evaluation and enthusiasts for playing non-native games, potentially shifting the Mac gaming conversation towards broader developer adoption.
Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future
3 months ago
- Apple's Mac mini and Mac Studio are favored for running AI agents due to demand for controlled, isolated systems that run continuously.
- AI tools being Mac-first or Mac-only strengthens Mac's position among developers, including those at frontier AI labs.
- Agentic AI is viewed as a whole-chip problem, leveraging Apple silicon's strengths beyond just GPU for tasks like tool-calling.
- Apple's AI strength traces back to early chip decisions, including the Neural Engine and neural accelerators in CPU and GPU.
- A shift toward local AI is driven by privacy, security, and cost concerns, with a hybrid future envisioned for on-device and cloud tasks.
- Transparent AI on iPhone and iPad includes features like Draw Things and SwingVision that operate seamlessly without overt AI labeling.
- AI development is progressing rapidly, with unpredictable advancements expected in the near future.
Perform DFU Restores on Apple Silicon Macs with Macvdmtool (2021)
3 months ago
- macvdmtool is a tool developed by marcan under the AsahiLinux project that simplifies putting Apple Silicon or Intel Macs with T2 chip (2018 and later) into DFU mode using a terminal command.
- Prerequisites include two Macs: a functional host Mac (Apple Silicon-based) and a target Mac that powers on and boots to question mark or recovery mode.
- Setup involves installing Xcode command-line tools, Apple Configurator 2, cloning and compiling macvdmtool, copying it to path, and connecting the Macs via a compatible USB-C or TB3/TB4 cable.
- To enter DFU mode, run 'sudo macvdmtool dfu' on the host Mac, then restore using cfgutil, Apple Configurator 2, or Finder, with IPSW files available from sources like Mr. Macintosh's database.
- Future automation may include scripts for downloading latest IPSWs and integrating with tools like Munki, with further resources and support available via GitHub and MacAdmins Slack.
Asahi Linux 7.1 Progress Report
3 months ago
- Linux 7.1 released with progress on M3, Apple bugs, and other updates.
- macOS 27 beta broke Asahi Linux boot; fix involves setting APFS boot flag.
- macOS 27 SMC firmware change caused battery driver issues; patched in kernel 7.0.12.
- M3 machines now support audio output, CPU frequency switching, and big.LITTLE scheduling.
- AVD firmware reverse-engineered for custom video decoder driver; AVC support working.
- m1n1 version 1.6.0 requires Rust for stage 2 builds and improves M3 support.
- New contributors helped with audio, AVD, and m1n1 development.

apple silicon

Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)

Distributing LLM Inference in DwarfStar

M-Chips: M7 with up to 1.5 TB – and why Apple is skipping the M6

Mac gaming is finally getting the overpowered upgrade it deserves

Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

Perform DFU Restores on Apple Silicon Macs with Macvdmtool (2021)

Asahi Linux 7.1 Progress Report