The problem

An invoicing app sends 20 000 payment reminders every morning at 9:00. A cron line on one server starts the script that sends them.

Last Tuesday that server rebooted at 8:59 for a kernel update, and nobody got a reminder that day. The week before, someone ran the script again after a timeout, and 3 000 customers got the same reminder twice.

The team wants one scheduler for all its timed work: it runs each job at its time, once, and a dead server delays jobs without losing them.