job queues

Job queues are deceptively tricky

2 months ago

Job queues, which schedule and run batch jobs, appear simple but involve complex details.
Three lenses for system design: wariness of queues, explicit limits, and fault models.
A practical example involves packing reference repos with two methods: wholesale (7 hours) and incremental (2 hours).
A proposed schedule runs wholesale on weekends and incremental on weekdays to balance size and up-to-dateness.
Job queue semantics for overlapping jobs include Parallel Spawn, Prefer New, Wait, and Prefer Old.
For the weekend workload with a 3-hour interval and 7-hour jobs, Prefer Old semantics work best to avoid waste.
Without Prefer Old, alternative solutions like cron-like scheduling or separate jobs are needed.
Poorly managed queues can lead to issues like unbounded growth, requiring manual fixes and hacks.
Designing with queues, limits, and fault models in mind leads to more robust and graceful degradation.
Understanding a system's assumptions helps users evaluate its fit for their workloads and avoid pitfalls.