Opsqueue: Lightweight batch processing queue for heavy loads – now open-source

9 months ago

Open-source release of opsqueue, an opinionated queueing system for batch processing operations.
Designed for scenarios where operations are generated by one system and executed by another, optimizing for throughput and parallel processing.
Introduces the 'generate-execute' pattern, a specialized version of producer-consumer, focusing on operations that take milliseconds to seconds.
Addresses limitations of existing queueing systems like Redis and Postgres by using SQLite for metadata and object storage for data, enhancing scalability and fault tolerance.
Allows consumers to decide the order of operation processing, offering flexibility in execution strategies like datacenter affinity or user fairness.
Utilizes chunking to reduce coordination overhead, with chunk size adjustable based on operation duration to balance parallelism and communication efficiency.
Integrated OpenTelemetry Tracing for monitoring, showing sub-millisecond overhead in queue communication.
Core implemented in Rust for performance and reliability, with Python client support via FFI bindings.
Proven in production, handling over 1,000,000 operations per hour with minimal overhead.
Encourages adoption for new projects or replacing existing ad-hoc solutions facing scalability issues.

Hasty Briefsbeta