The problem

3:04 a.m. The power supply of the PostgreSQL primary of an online shop fails. The two replicas are fine. The 12 app servers still point at the dead machine, and every checkout fails with a connection error.

The on-call engineer wakes up at 3:11, logs in at 3:19 and promotes a replica by hand at 3:26. The shop took no orders for 22 minutes, and nobody knows if the last orders before the crash reached the replica.

The shop wants the next failover to happen with no person involved, in under 30 seconds.