Indexing the Data Lake for Online Point Queries
4 hours ago
- Companies like Spotify need low-latency access to large datasets for online services and AI agents, but distributed SQL engines add seconds of overhead for point queries.
- Random Access Parquet (RAP) bridges the gap by using an external index to map keys directly to file locations, enabling precise ranged reads without scanning.
- RAP operates on existing Parquet files, eliminating the need for separate copies in key-value stores.
- The external index is a definitive multimap that returns exact file and row numbers for a given key, reducing dependent reads.
- Optimizations like sorting by key, co-grouping, coarser partitioning, one-page-per-key, ZSTD frame resets, blobs, interleaving columns, storage alignment, and covering indexes further reduce latency and bandwidth.
- These optimizations trade off some batch analytics benefits for faster point queries.
- RAP makes historical and long-tail data viable for interactive access, transforming the data lake into a dual-purpose storage for analytical and interactive workloads.