A distributed queue in a single JSON file on object storage

4 days ago

Replaced internal indexing job queue with a new version using a single queue file on object storage for better performance.
New queue design features FIFO execution, at-least-once guarantees, and 10x lower tail latency.
Object storage chosen for its simplicity, predictability, scalability, and ease of maintenance.
Initial design involved a single file (queue.json) with CAS operations for atomic updates.
Group commit technique introduced to batch writes and improve throughput.
Stateless broker introduced to eliminate contention over the queue object, handling all interactions with object storage.
High-availability ensured through broker failover and job heartbeats to detect and recover from failures.
Final system is reliable, scalable, and handles high throughput with at-least-once delivery guarantees.

Hasty Briefsbeta