on-device ai

Bilingual

Why on-device agentic AI can't keep up
2 months ago
- Local on-device AI offers privacy, zero latency, and no API costs, but is largely impractical for most consumer devices.
- Consumer hardware faces RAM constraints, with 8-16GB common, leaving only 4-8GB for AI models after OS and apps.
- A capable 7B parameter model requires ~5GB, but agentic tasks also need large KV cache, quickly exceeding 8GB RAM.
- Even small agentic tasks need at least 32K token context, requiring ~16GB RAM just for AI on devices.
- Rising RAM prices and competition with datacenter HBM worsen memory availability for consumer devices.
- On-device inference speed drops drastically with context size, e.g., from 100 to <10 tok/s at 16K context on a desktop GPU.
- Battery and heat issues make sustained on-device agentic AI use impractical on phones.
- Cloud inference avoids these limits but would require enormous datacenter capacity for billions of devices.
- Realistic agentic AI on consumer devices remains years away without major hardware or architecture breakthroughs.
Show HN: Smriti: Your Photos, Your Memories
2 months ago
- Smriti indexes photos in place on your drive without copying or moving originals.
- It offers on-device face recognition grouping, map plotting of geotagged photos, and unified search across people, places, dates, and albums.
- Smriti does not require an account, cloud subscription, or server; it operates with zero background processes when closed.
- Features include 'this day N years ago' memories, duplicate/near-duplicate detection, and optional local visual search.
- The assistant can search and draft albums using Smriti's own tools but does not modify or delete originals.
- Smriti stores its database on the same drive as photos, making it portable across devices.
- It respects privacy with no telemetry, analytics, or anonymous usage statistics; all processing stays on-device.
- Optional features like face recognition and offline geocoding can be declined without affecting core functionality.
- Smriti checks for updates only if opted in, with no photo data or metadata sent.
- Licensed under Apache-2.0, free forever, with no sign-up or telemetry.
DepthART: Scaling Foundation Monocular Depth to Tiny Models
2 months ago
- Recent geometric foundation models have improved monocular depth estimation, but their benefits are limited for tiny models.
- DepthART is a compact model designed for robust on-device depth estimation across diverse scenes.
- It uses bias-resistant data sampling and camera-conditioned fine-tuning to address dataset-specific overfitting and unstable metric adaptation.
- These methods improve cross-dataset generalization and metric depth prediction in capacity-constrained models.
- DepthART has been accepted to ACM Multimedia 2026.
Model 4: Our best model yet for local dictation and cleanup on Mac
2 months ago
- Epilude Model 4 is the fourth generation on-device model for Local Mode, offering the largest quality jump since its launch while maintaining the same 1.5 GB download size and speed, with no data leaving the user's Mac.
- Local Mode uses two models: a speech model to convert voice to text and a cleanup model to refine transcripts by removing filler words, fixing punctuation, handling self-corrections, and matching tone; Model 4 is a new fine-tuned cleanup model based on the Qwen family, focusing on dictation cleanup without changing the speech model.
- Evaluation is based on a zero-tolerance approach for critical errors that change meaning or leak instructions, using an internal benchmark of 90 dictation scenarios; Model 4 passes nine more scenarios than Model 3 and reduces critical meaning errors by 56%, excelling in mid-sentence self-corrections, dates, mixed-language speech, and casual tone.
- Key lessons from building Model 4 include that label quality is more important than data volume, as fixing mislabeled training examples improved self-correction, and weight averaging multiple model runs outperformed individual models by eliminating rare failure modes.
- Model 4 has limitations, such as struggling with long, rambling passages and spoken questions, where Cloud mode still leads in overall performance; it requires Apple Silicon and 16 GB of memory for full AI cleanup, but works offline with basic formatting on smaller machines, ensuring privacy by processing all data locally at no extra cost.
Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone
2 months ago
- Bonsai 27B is a new multimodal model from the Bonsai family, based on Qwen3.6 27B, and the first of its capability class to run on a phone.
- It comes in two variants: Ternary Bonsai 27B (5.9 GB) for quality on laptops and 1-bit Bonsai 27B (3.9 GB) for footprint on phones, both using low-bit weights end-to-end with no high-precision escapes.
- The model retains 95% (Ternary) and 90% (1-bit) of full-precision baseline performance across benchmarks, excelling in math, coding, and tool calling for agentic workloads.
- Bonsai 27B enables local execution for agentic AI, reducing costs and enhancing privacy by keeping data on-device, and supports hybrid deployments with cloud models.
- It achieves high speeds (up to 163 tok/s on RTX 5090) and fits phone memory constraints, with a 262K-token context, multimodal vision, and speculative decoding.
- The release represents a paradigm shift in intelligence density, advancing AI deployment on everyday devices under the Apache 2.0 License, with plans for larger models.
Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction
3 months ago
- Google researchers introduce a method to retrofit Multi-Token Prediction (MTP) onto frozen production models like Gemini Nano v3 to accelerate on-device inference without needing separate drafters.
- This approach addresses mobile constraints (energy, memory) by leveraging the main model's internal state via a lightweight MTP head, eliminating memory redundancy and reducing latency.
- MTP integration on Pixel 9 and 10 devices shows significant speedups (50%+) and energy savings in features like AI Notification Summaries and Proofread, while maintaining output compatibility.
- Future work includes exploring parallel decoding and branching techniques to handle language ambiguity more efficiently under strict mobile constraints.

on-device ai

Why on-device agentic AI can't keep up

Show HN: Smriti: Your Photos, Your Memories

DepthART: Scaling Foundation Monocular Depth to Tiny Models

Model 4: Our best model yet for local dictation and cleanup on Mac

Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction