2 months ago
- Cue, a voice-activated AI agent, reduced latency by 44% and increased dictation usage by 30% by running Gemma 4 E4B locally for text polishing.
- Local inference with Gemma 4 eliminated marginal costs for the polish step, enabling unlimited free dictation without paywalls.
- Gemma 4's on-device performance (median 488 ms) stays under the 500 ms threshold for a lag-free user experience, replacing a slower cloud-based pipeline.
- The polish pipeline corrects formatting, filler words, homophones, and adapts to context (e.g., app type) using a compact ~400-token system prompt.
- Cue inverted its original architecture: Gemma 4 local is now the default, with cloud as a fallback, after benchmarks showed it avoids over-editing while maintaining quality.
- The team is expanding Gemma's role into memory (learning user style) and local agentic tasks via function-calling, and plans to open-source its evaluation methodology.