Better Vector Search for Long Documents: Chunking Inside Manticore Search
3 hours ago
- - Manticore's auto-embedding feature now includes chunk_strategy option in table definitions to handle long documents that exceed the model's token limit.
- - Five chunking strategies are available: truncate (default), mean, fixed, recursive, and sentence, each producing either single or multiple vectors per document.
- - Chunking significantly improves recall for buried content: recall@5 increased from 55.1% to 83.3% and MRR from 0.44 to 0.70 in testing.
- - Smaller chunk sizes (128-256 tokens) generally improve search quality, while token overlap offers minimal benefit but increases vector count and RAM usage.
- - The feature works with local and remote embedding models, requires no separate pipeline or table for chunks, and integrates seamlessly with existing KNN search queries.