2 months ago
- Gemini Distillation Service trains a smaller 'student' model using outputs and reasoning from a larger 'teacher' model.
- It bridges the gap between frontier model capabilities and production efficiency, offering lower latency and cost.
- Unlike supervised fine-tuning (SFT), distillation leverages both final responses and internal reasoning paths from the teacher.
- Supported models: teacher gemini-3.1-pro, student gemini-2.5-flash.
- Ideal for high-volume, latency-sensitive applications, scenarios lacking ground-truth data, complex reasoning tasks, and when teacher outperforms student significantly.
- Setup requires allowlist access, enabling Agent Platform API, IAM admin role, and us-central1 region.
- Datasets are prompt-only JSONL files in Cloud Storage; minimum 1,000 examples recommended for quality improvement.
- Configuration includes candidateCount for teacher generation and hyperparameters like epochCount (1-100) and learningRateMultiplier (0.25-4).
- Jobs can be submitted via REST API, with support for continuous tuning from previously distilled checkpoints.
- Monitor through API or Google Cloud console (with known UI limitations during early access).
- After completion, an endpoint is created; evaluate by sending prediction requests and comparing to base and teacher models.
- Limitations: max 50K examples, 1GB file size, 8K input tokens, 24K output tokens, text-only, no CMEK, and hyperparameter ranges.
- Access granted via contacting support, with a default 30-day access period and concurrent quota of 4.