Day 30 10 min
When a dependency is down
1 / 6 · ≈ 14 min left
The problem
Black Friday, 10:02. The recommendation service of an online bookshop stops answering. It still accepts connections, but it sends nothing back.
Each product page waits for the "You may also like" block. After 2 seconds every worker thread of every app server is waiting on it. The product pages, the cart and the checkout all time out, though none of them needs recommendations to work.
The team wants the product page to load in under a second while the recommendation service is down.