Bilingual

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction
19 days ago
- Google researchers introduce a method to retrofit Multi-Token Prediction (MTP) onto frozen production models like Gemini Nano v3 to accelerate on-device inference without needing separate drafters.
- This approach addresses mobile constraints (energy, memory) by leveraging the main model's internal state via a lightweight MTP head, eliminating memory redundancy and reducing latency.
- MTP integration on Pixel 9 and 10 devices shows significant speedups (50%+) and energy savings in features like AI Notification Summaries and Proofread, while maintaining output compatibility.
- Future work includes exploring parallel decoding and branching techniques to handle language ambiguity more efficiently under strict mobile constraints.
Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone
7 days ago
- Bonsai 27B is a new multimodal model from the Bonsai family, based on Qwen3.6 27B, and the first of its capability class to run on a phone.
- It comes in two variants: Ternary Bonsai 27B (5.9 GB) for quality on laptops and 1-bit Bonsai 27B (3.9 GB) for footprint on phones, both using low-bit weights end-to-end with no high-precision escapes.
- The model retains 95% (Ternary) and 90% (1-bit) of full-precision baseline performance across benchmarks, excelling in math, coding, and tool calling for agentic workloads.
- Bonsai 27B enables local execution for agentic AI, reducing costs and enhancing privacy by keeping data on-device, and supports hybrid deployments with cloud models.
- It achieves high speeds (up to 163 tok/s on RTX 5090) and fits phone memory constraints, with a 262K-token context, multimodal vision, and speculative decoding.
- The release represents a paradigm shift in intelligence density, advancing AI deployment on everyday devices under the Apache 2.0 License, with plans for larger models.
Model 4: Our best model yet for local dictation and cleanup on Mac
a day ago
- Epilude Model 4 is the fourth generation on-device model for Local Mode, offering the largest quality jump since its launch while maintaining the same 1.5 GB download size and speed, with no data leaving the user's Mac.
- Local Mode uses two models: a speech model to convert voice to text and a cleanup model to refine transcripts by removing filler words, fixing punctuation, handling self-corrections, and matching tone; Model 4 is a new fine-tuned cleanup model based on the Qwen family, focusing on dictation cleanup without changing the speech model.
- Evaluation is based on a zero-tolerance approach for critical errors that change meaning or leak instructions, using an internal benchmark of 90 dictation scenarios; Model 4 passes nine more scenarios than Model 3 and reduces critical meaning errors by 56%, excelling in mid-sentence self-corrections, dates, mixed-language speech, and casual tone.
- Key lessons from building Model 4 include that label quality is more important than data volume, as fixing mislabeled training examples improved self-correction, and weight averaging multiple model runs outperformed individual models by eliminating rare failure modes.
- Model 4 has limitations, such as struggling with long, rambling passages and spoken questions, where Cloud mode still leads in overall performance; it requires Apple Silicon and 16 GB of memory for full AI cleanup, but works offline with basic formatting on smaller machines, ensuring privacy by processing all data locally at no extra cost.

Hasty Briefsbeta

#on-device ai

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction

Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone

Model 4: Our best model yet for local dictation and cleanup on Mac