How to Design a Distributed Job Scheduler
Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single "scheduler" box, and that box becomes a single point of failure.
Useful takeaways from this story.
Now the same cron line lives on every box, so the invoice job fires ten times instead of once.
Move the cron to a single "scheduler" box, and that box becomes a single point of failure.
Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it.
Building the complete brief
The page is ready to read now. The fuller skim-friendly version will appear here automatically.
The useful part
Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single "scheduler" box, and that box becomes a single point of failure. Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it.
What to take from it
The trouble starts when one server becomes ten.
Details worth keeping
Almost every backend eventually needs to run code on a schedule. Generate the weekly report every Monday at 7 AM. Any job due during that downtime window silently never fires.
Keep reading in the app
Open the app view to save this story, compare related coverage, and continue from the same source.