2 months ago
- Job queues, which schedule and run batch jobs, appear simple but involve complex details.
- Three lenses for system design: wariness of queues, explicit limits, and fault models.
- A practical example involves packing reference repos with two methods: wholesale (7 hours) and incremental (2 hours).
- A proposed schedule runs wholesale on weekends and incremental on weekdays to balance size and up-to-dateness.
- Job queue semantics for overlapping jobs include Parallel Spawn, Prefer New, Wait, and Prefer Old.
- For the weekend workload with a 3-hour interval and 7-hour jobs, Prefer Old semantics work best to avoid waste.
- Without Prefer Old, alternative solutions like cron-like scheduling or separate jobs are needed.
- Poorly managed queues can lead to issues like unbounded growth, requiring manual fixes and hacks.
- Designing with queues, limits, and fault models in mind leads to more robust and graceful degradation.
- Understanding a system's assumptions helps users evaluate its fit for their workloads and avoid pitfalls.