Be on call for real outages, ten minutes a day.
Each chapter is a real incident: you decide what to do at each step, then read what the team found.
Today · day 1 · 10 min
AWS S3 2017: the command that removed too many servers
0 of 40 chapters done
The path
Changes and config
- AWS S3 2017: the command that removed too many servers 1
- Cloudflare 2019: the WAF rule that used all the CPU 2
- Google Compute Engine 2016: the config push that withdrew every IP block 3
- Cloudflare 2022: the BGP policy change that took 19 data centers offline 4
- Knight Capital 2012: the old code left on one of eight servers 5
- Atlassian 2022: the script that deleted 883 sites 6
- Google Cloud 2025: the quota policy with blank fields 7
- Cloudflare 2025: the Bot Management feature file that doubled in size 8
Data
- GitLab 2017: the deleted database directory 9
- Sentry 2015: PostgreSQL transaction ID wraparound 10
- Mandrill 2019: the shard that stopped taking writes 11
- GoCardless 2017: the database cluster that did not fail over 12
- Roblox 2021: the 73-hour Consul outage 13
- Let's Encrypt 2020: the CAA recheck that checked one name 14
- Anthropic 2025: three bugs that degraded Claude's answers 15
- Stripe 2019: a shard that could not elect a primary 16
Networks and DNS
- Facebook 2021: the backbone that took its own DNS offline 17
- Slack 2021: an overloaded Transit Gateway on the first workday of the year 18
- AWS us-east-1 2025: a race in DynamoDB's DNS automation 19
- OpenAI 2024: a telemetry service that overloaded the Kubernetes control planes 20
- Reddit 2023: the Kubernetes upgrade that removed a node label 21
- Cloudflare 1.1.1.1 2025: the resolver prefixes withdrawn by a test change 22
- Datadog 2023: the automatic OS update that cut nodes off the network 23
- Cloudflare 2020: the Atlanta router change that pulled in the backbone’s traffic 24
Capacity and cascades
- AWS Kinesis 2020: the capacity addition that hit an OS thread limit 25
- Google 2020: the User ID Service that ran out of quota 26
- Cloudflare 2023: the data center that lost all power 27
- Joyent 2014: every server in us-east-1 rebooted at once 28
- AWS 2021: congestion on the internal network in us-east-1 29
- Heroku 2025: the OS update that cut dyno networking 30
- Monzo 2017: linkerd, Kubernetes and a payments outage 31
- Square 2017: retries that overloaded merchant authentication 32
Security
- CircleCI 2023: a stolen session and every customer secret 33
- Heroku 2022: stolen GitHub OAuth tokens 34
- Codecov 2021: a modified Bash Uploader in customers’ CI 35
- npm event-stream 2018: the new maintainer and the hidden payload 36
- npm left-pad 2016: the unpublished package that broke builds 37
- Cloudflare 2023: the credentials left over from the Okta breach 38
- LastPass 2022: a developer's home computer and the backup keys 39
- Matrix.org 2019: the outdated Jenkins and the forwarded SSH keys 40