Frame selection is the whole game: notes on making LLMs watch video
15 hours ago
- Using video over text lets the LLM see the raw event, not a human's compressed version, catching details like error messages or editing cuts.
- Uniform sampling (e.g., one frame per second) fails by both missing rare events and wasting tokens on near-identical frames.
- Token budget is tight: only about 100–150 frames per video, so each frame must earn its place.
- Scene detection uses ffmpeg's scene score with a rolling average threshold to adapt to animation or slow pans without manual tuning.
- Deduplication compares 16×16 RGB signatures against a sliding window of last four kept frames, plus a percentage threshold that handles small subjects via a 32×32 grid and hard change detection.
- A third channel for local changes (e.g., text/UI) uses a 192×192 signature, strict tolerance, and a cooldown to avoid budget waste.
- Text extraction uses embedded subtitles or local Whisper, with manual subtitle alignment as a fallback.
- Output is plain JPEGs, a transcript, and a manifest file, making it model-agnostic and portable.
- An MCP server allows any client to call `watch_video` and get fused results, with caching and resizing to 768px.
- The tool is free, MIT-licensed, and runs locally: `pip install claude-real-video[whisper]` and `crv <url>`.