Apple researchers develop on-device AI agent that interacts with apps
4 days ago
- #AI
- #Machine Learning
- #User Interface
- Ferret-UI Lite is a lightweight model with 3 billion parameters that matches or surpasses larger models up to 24 times its size.
- It is part of the Ferret family, which includes Ferret-UI and Ferret-UI 2, designed for enhanced mobile UI understanding.
- Ferret-UI Lite uses real and synthetic training data, on-the-fly cropping and zooming techniques, and supervised fine-tuning.
- The model compensates for its small size by cropping and re-predicting on specific GUI segments.
- It generates its own training data using a multi-agent system that interacts with live GUI platforms.
- Ferret-UI Lite was trained and evaluated on Android, web, and desktop GUI environments.
- It performs well on short-horizon tasks but struggles with complex, multi-step interactions.
- The model offers local and private GUI interaction, avoiding cloud processing.