Frame selection is the whole game: notes on making LLMs watch video

15 hours ago

Using video over text lets the LLM see the raw event, not a human's compressed version, catching details like error messages or editing cuts.
Uniform sampling (e.g., one frame per second) fails by both missing rare events and wasting tokens on near-identical frames.
Token budget is tight: only about 100–150 frames per video, so each frame must earn its place.
Scene detection uses ffmpeg's scene score with a rolling average threshold to adapt to animation or slow pans without manual tuning.
Deduplication compares 16×16 RGB signatures against a sliding window of last four kept frames, plus a percentage threshold that handles small subjects via a 32×32 grid and hard change detection.
A third channel for local changes (e.g., text/UI) uses a 192×192 signature, strict tolerance, and a cooldown to avoid budget waste.
Text extraction uses embedded subtitles or local Whisper, with manual subtitle alignment as a fallback.
Output is plain JPEGs, a transcript, and a manifest file, making it model-agnostic and portable.
An MCP server allows any client to call `watch_video` and get fused results, with caching and resizing to 768px.
The tool is free, MIT-licensed, and runs locally: `pip install claude-real-video[whisper]` and `crv <url>`.

LQIPs are small images that load quickly, providing immediate visual feedback while full-resolution images load.
Handsum is a new LQIP file format based on Discrete Cosine Transform (DCT) like Blurhash, Thumbhash, and JPEG.
Handsum files have fixed sizes: 48 bytes at lowest quality and 147 bytes at highest, enabling fixed-size database columns.

Blick enables instant editing with no import or project setup dialogs, opening directly to a blank project with autosave.
Users can start exports and continue editing without interruptions; even fix mistakes mid-render without restarting.
Keyframing is done directly in the timeline, allowing live animation recording during playback.

MiniMax H3 is an open-weights omni-modal video model that can generate video up to 2K resolution and 15 seconds with native stereo audio.
It supports text-to-video, image-to-video, first-and-last-frame control, and reference-to-video using images, video, or audio.
The model integrates multimodal context understanding, collapsing multiple tasks into one by processing images, audio, and video together.

JPEG files can be saved with low frequency components first, allowing partially downloaded images to display at low resolution instead of being cut off.
This is achieved through multiple scans in the file, each with a header; the first scan includes the lowest DC Fourier bin for all three color channels.
Scans define spectral ranges and precision, enabling construction of JPEGs where future scans overwrite already rendered data.

TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.

LQIPs are small images that load quickly, providing immediate visual feedback while full-resolution images load.
Handsum is a new LQIP file format based on Discrete Cosine Transform (DCT) like Blurhash, Thumbhash, and JPEG.
Handsum files have fixed sizes: 48 bytes at lowest quality and 147 bytes at highest, enabling fixed-size database columns.

Blick enables instant editing with no import or project setup dialogs, opening directly to a blank project with autosave.
Users can start exports and continue editing without interruptions; even fix mistakes mid-render without restarting.
Keyframing is done directly in the timeline, allowing live animation recording during playback.

MiniMax H3 is an open-weights omni-modal video model that can generate video up to 2K resolution and 15 seconds with native stereo audio.
It supports text-to-video, image-to-video, first-and-last-frame control, and reference-to-video using images, video, or audio.
The model integrates multimodal context understanding, collapsing multiple tasks into one by processing images, audio, and video together.

JPEG files can be saved with low frequency components first, allowing partially downloaded images to display at low resolution instead of being cut off.
This is achieved through multiple scans in the file, each with a header; the first scan includes the lowest DC Fourier bin for all three color channels.
Scans define spectral ranges and precision, enabling construction of JPEGs where future scans overwrite already rendered data.

TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.

Hasty Briefsbeta