The problem

Every night at 2 a.m., 40 sellers on a marketplace push their price and stock changes through an API. Each change is a job in a queue. Workers take the jobs and write them into the search index.

For 30 minutes, the sellers send 5 000 jobs per second. The workers finish 3 000. The queue gains 2 000 jobs every second, and a buyer at 2:30 a.m. still sees yesterday's price.

Someone has to decide what happens to job number 5 000 in a second that only has room for 3 000. If nobody decides, the queue grows until the broker runs out of memory or disk.