2 months ago
- Gemini Distillation Service trains a smaller student model using outputs and reasoning patterns from a larger teacher model, enabling lower latency and cost while maintaining deep reasoning.
- Distillation uses teacher responses and raw thoughts, unlike supervised fine-tuning which only uses final text output.
- Supported models: teacher gemini-3.1-pro and student gemini-2.5-flash during early access.
- Recommended for high-volume latency-sensitive apps, lack of ground-truth data, complex reasoning tasks, and significant performance gaps between teacher and student.
- Prerequisites: project allowlist access, enable Agent Platform API, IAM role aiplatform.admin, and run jobs in us-central1 region.
- Dataset must be in JSONL format with prompt-only entries; no ground-truth answers needed; at least 1,000 examples recommended.
- Configure teacher generation with candidateCount (1-5, default 4) and student hyperparameters: epochCount (1-100, default 4) and learningRateMultiplier (0.25-4, default 1).
- Submit jobs via REST API to Agent Platform; supports both new jobs and continuous tuning from previous distilled checkpoints.
- Monitor jobs via API or console; console has known UI limitations for teacher sampling progress and epoch display.
- Evaluate by locating the registered endpoint, sending prediction requests, and comparing outputs against base and teacher models.
- Limitations: max 50,000 examples, 1 GB file size, 8,000 input tokens per entry, 24,000 output tokens, text-only, no CMEK, one-step distillation only.
- Request access via cloud-ai-tuning-service-support@google.com; default concurrent quota of 4; access initially for 30 days.