Lance table format explained simply, stupid (Animated)
3 months ago
- #big-data
- #file-format
- #AI
- Lance is a file format, table format, and catalog spec optimized for random reads and supports ad-hoc column additions without data copying.
- Key features of Lance include performance optimization for random reads (WHERE id = 123) while maintaining sequential read performance.
- Lance table format supports indexes like BTree, inverted index (FTS), and vectors (e.g., HNSW).
- Competing technologies like vortex by SpiralDB emerge due to the demand for multi-modal data lakes driven by AI.
- Significant 2025 big data updates include Iceberg V3 with VARIANT, turbopuffer's vector search, Apache Fluss for Flink streams, and acquisitions by Datadog and Databricks.