Oban, the job processing framework from Elixir, has come to Python
10 days ago
- #postgresql
- #python
- #job-queue
- Oban is a job processing tool used in Elixir and Python, leveraging PostgreSQL for job management.
- Oban-py comes in two versions: Open Source (OSS) and Pro, with Pro offering features like bulk operations and true parallelism.
- Jobs are inserted into the `oban_jobs` table and processed via PostgreSQL's `NOTIFY` and `FOR UPDATE SKIP LOCKED` for concurrency.
- Background processes in Oban include leader election, rescuing orphaned jobs, and pruning old jobs.
- Leader election is handled via PostgreSQL with a TTL-based lease, avoiding complex consensus protocols.
- Job retries use jittery exponential backoff to prevent thundering herds and ensure efficient retry scheduling.
- Oban-py is concurrent but not parallel in its OSS version, making Pro ideal for CPU-bound tasks.
- The codebase is praised for its clarity and structure, making it easy to understand and maintain.