Building a Robust Ingestion System for Any File of Any Size

8 hours ago

Mixedbread rebuilt its ingestion pipeline to handle arbitrarily large files by splitting processing into a slicer and parser.
The slicer cuts files into bounded slices based on type-specific units (pages, seconds, characters).
The parser processes one slice at a time on fixed-size workers, ensuring memory usage depends on slice size, not file size.
Key challenges include large file sizes, unpredictable processing costs from small files, and unsanitized files.
Slicing avoids fixed-size splitting, preserving semantic boundaries to maintain retrieval quality.
Processing involves probing file extent, slicing, parsing, baton-passing between workers, and completion.
Slice sizes are chosen to fit within worker constraints, making the system retryable and idempotent.
Semantic boundaries are preserved by adjusting slice starts based on parsed chunk ends.
Memory is bounded via streaming, render limits, and lazy chunk iteration.
Cost estimation before ingestion uses cheap indicators like page count or duration for quotas.
Future improvements aim to enable concurrent slice processing while preserving semantic integrity.

Hasty Briefsbeta