HorizonDB, a geocoding engine in Rust that replaces Elasticsearch
16 days ago
- #geospatial
- #database
- #Rust
- Radar processes over 1 billion API calls per day, serving hundreds of millions of devices globally.
- HorizonDB, a geospatial database written in Rust, was developed to consolidate multiple location services into a single, high-performance binary.
- Prior to HorizonDB, geocoding was split across Elasticsearch, microservices, and MongoDB, leading to operational and scaling challenges.
- HorizonDB utilizes technologies like RocksDB, S2, Tantivy, FSTs, LightGBM, and FastText to enhance performance and efficiency.
- Rust was chosen for its safety, performance, and concurrency features, making it ideal for systems programming.
- RocksDB provides microsecond response times, serving as the primary record store with high efficiency.
- S2 enables efficient spatial indexing, converting O(n) point-in-polygon lookups into constant time operations.
- FSTs offer efficient string compression and prefix queries, caching millions of 'happy paths' with minimal memory usage.
- Tantivy, an in-process inverted index, was preferred over external services like Elasticsearch for its simplicity and performance.
- FastText improves search precision by representing words as numeric vectors, handling typos and out-of-vocabulary words effectively.
- LightGBM models classify query intent, structuring queries to enhance search performance and precision.
- Apache Spark processes hundreds of millions of data points quickly, with near-linear scalability.
- HorizonDB has significantly improved cost, performance, and scalability for Radar's geolocation services.
- Radar is hiring engineering talent and invites interested candidates to check their jobs page.