Glossary

active-active: A service runs and serves traffic in several datacenters at once, so one datacenter can fail while the others keep answering.; Square 2017: retries that overloaded merchant authentication
Admission webhook: An HTTP call the API server makes to another service before it accepts a write. If that service does not answer, the write times out.; Reddit 2023: the Kubernetes upgrade that removed a node label
Anycast: Announcing the same prefix from many data centers, so each user reaches the nearest one. Cloudflare serves almost all its services this way.; Cloudflare 1.1.1.1 2025: the resolver prefixes withdrawn by a test change
API key: A secret string that a script sends to a service, here Cloudflare, to make changes without a person logging in.; Matrix.org 2019: the outdated Jenkins and the forwarded SSH keys
apiserver: The Kubernetes process that every other component asks for the cluster’s state. It reads and writes that state in etcd.; Monzo 2017: linkerd, Kubernetes and a payments outage
Approximate top-k: A fast operation that finds the k highest values in an array and may be wrong on the lowest ones it returns. JAX's version, jax.lax.approx_max_k, has a default recall_target of 0.95.; Anthropic 2025: three bugs that degraded Claude's answers
Authoritative name server: The DNS server that holds the answers for a domain. Facebook's run at fixed IP addresses that Facebook advertises with BGP.; Facebook 2021: the backbone that took its own DNS offline
Authorization: Boulder's record that an account proved control of one domain name. Let's Encrypt reuses it for 30 days.; Let's Encrypt 2020: the CAA recheck that checked one name
authorized_keys: The file on a server that lists the SSH public keys allowed to log in to an account. By default, sshd also reads the legacy file authorized_keys2.; Matrix.org 2019: the outdated Jenkins and the forwarded SSH keys
Backbone: The network Facebook built to connect all its computing facilities, with fiber links between every data center.; Facebook 2021: the backbone that took its own DNS offline
Backbone: Cloudflare’s private lines between its data centers. Traffic between two sites can use them instead of the public Internet.; Cloudflare 2020: the Atlanta router change that pulled in the backbone’s traffic
Backoff: A client waits longer after each failed try before it retries. Without it, retries add load to a system that is already failing.; AWS 2021: congestion on the internal network in us-east-1
backoff: A wait before each retry, usually longer after each failure, so retries do not hit the server as fast as the client can send them.; Square 2017: retries that overloaded merchant authentication
Backtracking: What a regex engine like PCRE does when a match fails: it goes back and tries another way to split the input. Some patterns make it try a very large number of ways.; Cloudflare 2019: the WAF rule that used all the CPU
Baseline Requirements (BRs): The rules every publicly trusted certificate authority must follow, including when to check CAA and how fast to revoke.; Let's Encrypt 2020: the CAA recheck that checked one name
Bash Uploader: Codecov’s shell script that detects the CI settings, gathers coverage reports and uploads them to Codecov.; Codecov 2021: a modified Bash Uploader in customers’ CI
BGP: The protocol networks use to announce which IP blocks they can reach. Google announces each GCE IP block from several sites, so users take the shortest path.; Google Compute Engine 2016: the config push that withdrew every IP block
BGP: Border Gateway Protocol: how networks on the Internet tell each other which IP address ranges they can reach.; Facebook 2021: the backbone that took its own DNS offline
BGP withdrawal: A BGP message that tells peers a route to a prefix is no longer available. Once every route is withdrawn, packets to that prefix have nowhere to go.; Cloudflare 1.1.1.1 2025: the resolver prefixes withdrawn by a test change
C2: Command and control: a program on a hacked machine that connects to the attacker's server and runs the commands it receives.; Cloudflare 2023: the credentials left over from the Okta breach
CAA record: A DNS record where a domain owner lists the certificate authorities allowed to issue certificates for that domain.; Let's Encrypt 2020: the CAA recheck that checked one name
Canary step: Deploying a change at one site first and checking that the site still works before the change goes further.; Google Compute Engine 2016: the config push that withdrew every IP block
Caret range: A version rule like ^0.1.0 in package.json. npm installs the newest version that matches it, so a new patch release reaches every fresh install.; npm event-stream 2018: the new maintainer and the hidden payload
Cell: One of several isolated copies of a service. Each copy serves only part of the requests, so a failure in one copy hits only that part.; AWS S3 2017: the command that removed too many servers
Cell: One of several isolated copies of a service. Each copy handles a subset of the requests, so a failure stays inside that copy.; AWS Kinesis 2020: the capacity addition that hit an OS thread limit
cgroup: A Linux feature that caps the CPU and memory a group of processes can use on a machine.; Square 2017: retries that overloaded merchant authentication
checksum: A short value computed from a file’s bytes, like a SHA-256 sum. Change one byte of the file and the sum changes.; Codecov 2021: a modified Bash Uploader in customers’ CI
CI environment variables: The variables a CI job runs with. Teams put tokens and keys there so the job can deploy or call other services.; Codecov 2021: a modified Bash Uploader in customers’ CI
Circuit breaker: A wrapper around a remote call that counts failures. Past a threshold it returns an error at once, without making the call.; Stripe 2019: a shard that could not elect a primary
Colocation rule: A Pacemaker rule that says two resources must, or must never, run on the same node.; GoCardless 2017: the database cluster that did not fail over
config var: A Heroku setting that an app reads as an environment variable. Customers keep secrets in config vars.; Heroku 2022: stolen GitHub OAuth tokens
Consul: HashiCorp's service registry and key/value store. At Roblox, 5 voter servers hold the state and 5 non-voters serve extra reads.; Roblox 2021: the 73-hour Consul outage
Control plane: The part of a Kubernetes cluster that manages its state. OpenAI uses it for cluster administration.; OpenAI 2024: a telemetry service that overloaded the Kubernetes control planes
Control plane: The nodes that run the Kubernetes API server and its etcd database. This cluster has three.; Reddit 2023: the Kubernetes upgrade that removed a node label
Control plane: The customer-facing interface for all Cloudflare services: the dashboard website and the APIs. Customers change their settings through it.; Cloudflare 2023: the data center that lost all power
Control plane: The SmartDataCenter services that run the API and the boot sequences. Some of its components are stateful: they keep data that they must get back after a restart.; Joyent 2014: every server in us-east-1 rebooted at once
Control plane: The APIs that create and manage AWS resources, such as launching an EC2 instance. Running instances keep working without it.; AWS 2021: congestion on the internal network in us-east-1
Core proxy: The Cloudflare system, called FL and in its new version FL2, that every request passes through. Bot Management is one of the modules it runs.; Cloudflare 2025: the Bot Management feature file that doubled in size
Credential rotation: Replacing a password, token or key with a new one and revoking the old one, so a copy of the old one stops working.; Cloudflare 2023: the credentials left over from the Okta breach
curl pipe to bash: Running a script straight from a URL, as in curl -s https://… | bash, without saving or checking it first.; Codecov 2021: a modified Bash Uploader in customers’ CI
Data plane: The nodes that run the workloads, such as model inference. They can keep running for a while without the control plane.; OpenAI 2024: a telemetry service that overloaded the Kubernetes control planes
Decryption key: The secret that turns encrypted data back into readable data. An attacker who copies an encrypted file without its key gets only unreadable bytes.; LastPass 2022: a developer's home computer and the backup keys
Disaster recovery site: A Cloudflare facility in Europe that can run critical control plane services when the Hillsboro data centers cannot.; Cloudflare 2023: the data center that lost all power
Distributed table: A ClickHouse table in the default database that queries the underlying tables, stored in the r0 database on each shard.; Cloudflare 2025: the Bot Management feature file that doubled in size
DNS Enactor: The component that applies a DNS plan in Amazon Route53, one transaction per endpoint. Three copies run independently, in three Availability Zones.; AWS us-east-1 2025: a race in DynamoDB's DNS automation
DNS Planner: The DynamoDB component that watches load balancer health and capacity and regularly writes a new DNS plan: a set of load balancers and weights for each endpoint.; AWS us-east-1 2025: a race in DynamoDB's DNS automation
Droplet lease: The lease that EC2's DropletWorkflow Manager (DWFM) holds on each physical server, or droplet. A droplet without an active lease cannot take new instances.; AWS us-east-1 2025: a race in DynamoDB's DNS automation
Dyno: A Heroku container where an app’s code runs.; Heroku 2025: the OS update that cut dyno networking
Election: The step where the remaining nodes of a shard choose a new primary after the old one fails.; Stripe 2019: a shard that could not elect a primary
endpoints: The list of pod IP addresses behind a Kubernetes Service. A Service scaled to 0 replicas has an empty list.; Monzo 2017: linkerd, Kubernetes and a payments outage
etcd: The distributed database where Kubernetes stores what is deployed and where. A write needs a majority of its nodes (a quorum).; Monzo 2017: linkerd, Kubernetes and a payments outage
Exact version: A dependency written as "0.0.3" with no ^ or ~. npm installs that version and no other.; npm left-pad 2016: the unpublished package that broke builds
exfiltration: Copying data out of a company’s systems to a place the attacker controls.; CircleCI 2023: a stolen session and every customer secret
Fail open: When a check fails, the service still serves the request instead of rejecting it.; Google Cloud 2025: the quota policy with blank fields
Feature file: The configuration file that lists the features (traits of a request) the Bot Management model uses. It is regenerated every few minutes and sent to every machine.; Cloudflare 2025: the Bot Management feature file that doubled in size
Feature flag: A switch that turns new code on gradually, region by region and project by project, starting with internal projects.; Google Cloud 2025: the quota policy with blank fields
Freelist: BoltDB's list of pages in its file that held deleted data and can be reused. The file itself never shrinks.; Roblox 2021: the 73-hour Consul outage
Front-end fleet: The Kinesis servers that handle authentication, throttling and routing of each request to the right shard. It has many thousands of servers.; AWS Kinesis 2020: the capacity addition that hit an OS thread limit
Global terminate: A mechanism built into Cloudflare that disables a single component on every server worldwide.; Cloudflare 2019: the WAF rule that used all the CPU
Grace period: A delay between a new quota decision and the moment the quota system enforces it.; Google 2020: the User ID Service that ran out of quota
hashed and salted: A password stored as the output of a one-way function, mixed with a random value per user, so equal passwords give different stored values.; Heroku 2022: stolen GitHub OAuth tokens
High availability cluster: The set of services that run in all three Hillsboro data centers at once and keep their data in sync. If one data center goes offline, the other two keep running them.; Cloudflare 2023: the data center that lost all power
Index subsystem: The S3 subsystem that holds the metadata and location of every S3 object in the region. Every GET, LIST, PUT and DELETE needs it.; AWS S3 2017: the command that removed too many servers
Internal network: The AWS network that hosts foundational services: monitoring, internal DNS, authorization and parts of the EC2 control plane. Customer applications do not run on it.; AWS 2021: congestion on the internal network in us-east-1
IP block: A contiguous group of internet addresses. Google uses IP blocks for GCE VMs, network load balancers and Cloud VPNs.; Google Compute Engine 2016: the config push that withdrew every IP block
IP rule: A routing policy rule that picks which route table a packet uses, for example by its source address. Cilium adds IP rules so pod traffic leaves through the right interface.; Datadog 2023: the automatic OS update that cut nodes off the network
Keylogger: Malware that records every key a person types, including passwords.; LastPass 2022: a developer's home computer and the backup keys
Kubernetes API server: The control plane component that exposes the Kubernetes HTTP API. Removing a service from a cluster goes through it.; OpenAI 2024: a telemetry service that overloaded the Kubernetes control planes
linkerd: A proxy on each node that routes and load balances calls between services. It learns service addresses from the Kubernetes apiserver.; Monzo 2017: linkerd, Kubernetes and a payments outage
local-preference: A BGP attribute shared between routers of the same network (iBGP). When two routes reach the same prefix, the route with the higher value wins.; Cloudflare 2020: the Atlanta router change that pulled in the backbone’s traffic
LVM snapshot: A copy of a disk volume at one moment, taken by the Linux volume manager. GitLab used these to copy production into staging.; GitLab 2017: the deleted database directory
machine account: An account used by software instead of a person, with its own token, for example to let one internal system call another.; Heroku 2022: stolen GitHub OAuth tokens
Maintainer: An npm account with the right to publish new versions of a package.; npm event-stream 2018: the new maintainer and the hidden payload
Master password: The one password a LastPass user types to open their vault. LastPass never stores it, and the vault key is derived from it on the user's device.; LastPass 2022: a developer's home computer and the backup keys
Maximum-prefix limit: A cap on how many prefixes a BGP session accepts from a peer. Above the cap, the router shuts the session down.; Cloudflare 2020: the Atlanta router change that pulled in the backbone’s traffic
MCP: Multi-Colo PoP, the architecture of Cloudflare's 19 busiest data centers. A layer of routers called spines connects the parts of each data center as a mesh.; Cloudflare 2022: the BGP policy change that took 19 data centers offline
Mixed precision: Some operations run in 16-bit floats (bf16) and others in 32-bit floats (fp32), so two results can differ in the last digits.; Anthropic 2025: three bugs that degraded Claude's answers
Multi-tenant data store: A database shared by many customers. Each shard holds the data of several tenants, isolated from each other.; Atlassian 2022: the script that deleted 883 sites
Multimog: Cloudflare's internal load balancer in MCP data centers, a variant of Unimog. It spreads requests across the servers of a data center.; Cloudflare 2022: the BGP policy change that took 19 data centers offline
Name squatting: Publishing under a package name that someone else used before, so that existing installs get your code.; npm left-pad 2016: the unpublished package that broke builds
Node label: A key and value on a Kubernetes node, such as node-role.kubernetes.io/master. Other resources pick nodes by matching their labels.; Reddit 2023: the Kubernetes upgrade that removed a node label
OAuth token: A token that lets CircleCI act on a customer’s GitHub account, for example to read their repositories.; CircleCI 2023: a stolen session and every customer secret
OAuth token: A token GitHub issues to an app like Heroku, so the app can act on a user’s repositories without the user’s password.; Heroku 2022: stolen GitHub OAuth tokens
optimistic transaction: A Redis transaction that WATCHes keys and aborts at EXEC if another client changed one of them. The client then runs it again.; Square 2017: retries that overloaded merchant authentication
OS thread limit: An operating system setting that caps how many threads a server may run. A process that needs more threads than the cap cannot create them.; AWS Kinesis 2020: the capacity addition that hit an OS thread limit
Out-of-band monitoring: Checks that run completely outside your own infrastructure and call your public APIs like a user does. They still alert when your own monitoring is down.; Datadog 2023: the automatic OS update that cut nodes off the network
Outdated data: A read that may miss the latest writes. The User ID Service rejects such reads "for security reasons".; Google 2020: the User ID Service that ran out of quota
Pacemaker: Cluster software that runs on each node, decides where each resource (a Postgres process, an IP address) runs, and moves it when a node fails.; GoCardless 2017: the database cluster that did not fail over
Panic: In Rust, a crash of the current thread. Calling unwrap() on a Result that holds an Err triggers one.; Cloudflare 2025: the Bot Management feature file that doubled in size
Parent and child orders: SMARS receives a "parent" order from another Knight system. It sends one or more "child" orders to venues, as needed based on the available liquidity.; Knight Capital 2012: the old code left on one of eight servers
Paxos: A protocol where a majority of processes must accept each proposal before it is committed. The User ID Service database uses it, with one leader process doing the writes.; Google 2020: the User ID Service that ran out of quota
Payload: The part of malicious code that does the harm, here encrypted so that a reader cannot see what it does.; npm event-stream 2018: the new maintainer and the hidden payload
PBKDF2: A function that turns a password into a key by hashing it many times. LastPass's default in 2022 was 100,100 iterations, so each guess costs an attacker 100,100 hashes.; LastPass 2022: a developer's home computer and the backup keys
Persistence: A way back in that an attacker sets up so that removing the first stolen account does not lock them out, such as a second account or an installed program.; Cloudflare 2023: the credentials left over from the Okta breach
pg_basebackup: The PostgreSQL tool that copies a running database cluster over a replication connection, used to start a new standby.; GitLab 2017: the deleted database directory
Placeholder package: A package that npm publishes under a removed name so that nobody else can take that name.; npm left-pad 2016: the unpublished package that broke builds
Placement subsystem: The S3 subsystem that allocates storage for new objects during a PUT. It needs the index subsystem to work.; AWS S3 2017: the command that removed too many servers
PMON: Knight's main risk monitoring tool. It shows positions after trades execute, and a person has to watch it: it sends no automated alerts.; Knight Capital 2012: the old code left on one of eight servers
Policy term: One named rule inside a BGP policy. The router evaluates the terms in order, and the first one that accepts or rejects a prefix decides.; Cloudflare 2022: the BGP policy change that took 19 data centers offline
Power Peg: An old SMARS function that Knight stopped using in 2003. Its code stayed on the servers and could still be called.; Knight Capital 2012: the old code left on one of eight servers
Prefix: A range of adjacent IP addresses that a network advertises to its peers over BGP. A prefix that is no longer advertised is "withdrawn" and cannot be reached.; Cloudflare 2022: the BGP policy change that took 19 data centers offline
Prefix: A block of IP addresses announced as one route, such as 1.1.1.0/24 (the 256 addresses from 1.1.1.0 to 1.1.1.255).; Cloudflare 1.1.1.1 2025: the resolver prefixes withdrawn by a test change
Prefix list: A named list of IP prefixes in a router config. A policy term that has prefix-list X in its from block only matches routes in that list.; Cloudflare 2020: the Atlanta router change that pulled in the backbone’s traffic
Primary: The one node of a shard that accepts writes. The other nodes copy its data.; Stripe 2019: a shard that could not elect a primary
Private Space: A dedicated Heroku environment that runs dynos and some add-ons in an isolated network.; Heroku 2025: the OS update that cut dyno networking
Progressive rollout: Pushing a change to a fraction of sites at a time, so that a new failure shows up before it reaches every site.; Google Compute Engine 2016: the config push that withdrew every IP block
provision-service: The Slack service that configures and tests every new instance before it serves traffic.; Slack 2021: an overloaded Transit Gateway on the first workday of the year
PXE boot: A server boots from the network: it asks a DHCP server for an address and downloads its boot image over TFTP. Joyent compute nodes boot this way.; Joyent 2014: every server in us-east-1 rebooted at once
Quota: The amount of a resource, here storage for the account database, that a service is allowed to use. Google sets quotas with automation.; Google 2020: the User ID Service that ran out of quota
Race condition: A bug that appears only when two processes act in one particular order. Here, two DNS Enactors worked on the same endpoint at nearly the same time.; AWS us-east-1 2025: a race in DynamoDB's DNS automation
Raft leader: The one voter that replicates every write to the other voters and decides when a write is committed. Consul uses the Raft algorithm to elect it.; Roblox 2021: the 73-hour Consul outage
Randomized exponential backoff: Each retry waits longer than the last, plus a random delay, so clients that failed together do not retry together.; Google Cloud 2025: the quota policy with blank fields
RCE: Remote code execution: a bug that lets someone on the network run their own code on your server.; Matrix.org 2019: the outdated Jenkins and the forwarded SSH keys
Replication lag: How far the standby is behind the primary, in WAL it has not yet replayed.; GitLab 2017: the deleted database directory
Revocation: The certificate authority marks a certificate as no longer valid before it expires. Clients that check revocation then refuse it.; Let's Encrypt 2020: the CAA recheck that checked one name
rotate: Replace a secret with a new one at the system it unlocks, and revoke the old one there.; CircleCI 2023: a stolen session and every customer secret
Route reflector: A Calico node that exchanges pod routes (over BGP) with every other node, so the other nodes only need to talk to the reflectors.; Reddit 2023: the Kubernetes upgrade that removed a node label
RPO: Recovery point objective: how much recent data a restore may lose, counted back from the failure.; Atlassian 2022: the script that deleted 883 sites
RTO: Recovery time objective: how quickly data must be recovered and returned to a customer.; Atlassian 2022: the script that deleted 883 sites
SERVFAIL: The DNS error a resolver returns when it cannot get an answer from a domain's name servers.; Facebook 2021: the backbone that took its own DNS offline
Service account: An account that a program uses instead of a person, for example the account a SaaS tool uses to read and write Jira tickets.; Cloudflare 2023: the credentials left over from the Okta breach
Service Control: The Google binary that checks whether each incoming API request is authorized and within quota. It reads quota and policy data from a regional datastore.; Google Cloud 2025: the quota policy with blank fields
Service discovery: How one service finds the address of another. At OpenAI it goes through Kubernetes DNS, which Kubernetes programs from the API server.; OpenAI 2024: a telemetry service that overloaded the Kubernetes control planes
Service Health Dashboard: The public AWS page (SHD) where AWS posts the status of each service in each region.; AWS S3 2017: the command that removed too many servers
Service Health Dashboard: The public AWS page where AWS posts the status of its services in each region.; AWS 2021: congestion on the internal network in us-east-1
Service topology: Cloudflare’s list of the data centers where a service’s prefixes should be announced. Data Localization Suite (DLS) services use it to stay inside one region.; Cloudflare 1.1.1.1 2025: the resolver prefixes withdrawn by a test change
Shard: One of several databases that each hold part of the keys. At Mandrill a hashing algorithm picks the shard for each key.; Mandrill 2019: the shard that stopped taking writes
Shard: One slice of a database cluster. Stripe splits data by kind into clusters and by quantity into shards, and each shard has several redundant nodes.; Stripe 2019: a shard that could not elect a primary
Shard-map: The cache each front-end server keeps of which back-end cluster owns which shard. Without a complete shard-map, a server cannot route requests.; AWS Kinesis 2020: the capacity addition that hit an OS thread limit
Single-user mode: A way to start the Postgres server outside normal operation. The wraparound shutdown does not apply there, so the docs ask for it to run VACUUM.; Sentry 2015: PostgreSQL transaction ID wraparound
SMARS: Knight's automated, high speed order router. It takes orders from other Knight systems and sends orders to exchanges and other trading venues.; Knight Capital 2012: the old code left on one of eight servers
Soft delete: A deletion that only marks data as deleted. The data stays recoverable until a retention period ends.; Atlassian 2022: the script that deleted 883 sites
Split brain: Two parts of a cluster each act as the only leader and accept writes. Their data then disagrees, and some of it gets lost.; Joyent 2014: every server in us-east-1 rebooted at once
SSH agent forwarding: An SSH option (ssh -A, or ForwardAgent yes) that lets a remote server use the private keys on your laptop for as long as you are logged in.; Matrix.org 2019: the outdated Jenkins and the forwarded SSH keys
SSO session: The cookie a single sign-on provider gives the browser after a login, including the 2FA step. Any browser that sends this cookie is treated as logged in.; CircleCI 2023: a stolen session and every customer secret
Staged rollout: Deploying a change to a few locations first and to the world last. Cloudflare software goes to DOG (staff only), then PIG (some free traffic), then 3 Canary PoPs, then everywhere.; Cloudflare 2019: the WAF rule that used all the CPU
Sticky routing: The load balancer sends the follow-up requests of a conversation to the server that served the first one.; Anthropic 2025: three bugs that degraded Claude's answers
Streaming: A feature of Consul 1.10 for sending updates across the cluster, designed to use less CPU and network bandwidth than long polling.; Roblox 2021: the 73-hour Consul outage
Synchronous replica: A replica the primary waits for: a write is confirmed to the client only after this replica has received it.; GoCardless 2017: the database cluster that did not fail over
systemd-networkd: The systemd service that configures network interfaces, routes and routing policy rules on a Linux host from .network files.; Datadog 2023: the automatic OS update that cut nodes off the network
Throttle: A limit on how much work a service takes at once. Joyent infrastructure has throttles "to ensure that it cannot run away with a machine".; Joyent 2014: every server in us-east-1 rebooted at once
Thundering herd: A service comes back, and all the requests that failed while it was down arrive at once and overload it.; Cloudflare 2023: the data center that lost all power
Top-p sampling: The model picks the next token at random, only among the most probable tokens whose probabilities add up to a threshold such as 0.99.; Anthropic 2025: three bugs that degraded Claude's answers
Transaction ID (XID): A 32-bit counter PostgreSQL assigns to each write. Rows store the XID that wrote them, and visibility rules compare XIDs.; Mandrill 2019: the shard that stopped taking writes
Transaction ID wraparound: The XID counter wraps around to zero, and rows written in the past suddenly look like they are in the future.; Sentry 2015: PostgreSQL transaction ID wraparound
Transit Gateway (TGW): An AWS network hub that interconnects VPCs. AWS manages and scales it, and Slack uses TGWs to link its VPCs.; Slack 2021: an overloaded Transit Gateway on the first workday of the year
Transitive dependency: A package your project installs because one of your dependencies needs it. You never listed it in your own package.json.; npm event-stream 2018: the new maintainer and the hidden payload
Unattended upgrade: An operating system package upgrade that a host installs on its own, with no operator starting it.; Heroku 2025: the OS update that cut dyno networking
Unattended upgrades: The Ubuntu feature that installs security updates on a timer, with no operator. Datadog’s hosts used the Ubuntu defaults: upgrades at 06:00 UTC plus up to 60 minutes of random delay.; Datadog 2023: the automatic OS update that cut nodes off the network
Unpublish: Removing a package version from the npm registry. In March 2016 an author could unpublish any version at any time.; npm left-pad 2016: the unpublished package that broke builds
VACUUM: The Postgres maintenance command that, among other work, marks old rows as frozen so they stay visible after wraparound. Autovacuum runs it automatically.; Sentry 2015: PostgreSQL transaction ID wraparound
VACUUM: The PostgreSQL command that, among other jobs, marks old rows as frozen so their XIDs no longer count. Autovacuum runs it in the background.; Mandrill 2019: the shard that stopped taking writes
Virtual IP (VIP): An IP address that Pacemaker moves between nodes. The Rails apps connect to the VIP, so they always reach the current primary.; GoCardless 2017: the database cluster that did not fail over
VPC: Virtual Private Cloud: an isolated network inside AWS. Slack runs its services in separate AWS accounts and dedicated VPCs.; Slack 2021: an overloaded Transit Gateway on the first workday of the year
WAF: Web application firewall: code on every Cloudflare server that checks each HTTP request against rules and blocks the ones that look like attacks.; Cloudflare 2019: the WAF rule that used all the CPU
WAL segment: A file of PostgreSQL's write-ahead log. A standby replays these files to stay in step with the primary.; GitLab 2017: the deleted database directory
Web tier: Slack's fleet of web servers. Autoscaling adds or removes instances based on CPU use and on busy Apache worker threads.; Slack 2021: an overloaded Transit Gateway on the first workday of the year
Wraparound: After about 2 billion new XIDs, an old XID would look like it is in the future. PostgreSQL stops assigning XIDs before that happens.; Mandrill 2019: the shard that stopped taking writes
XID: A PostgreSQL transaction ID: a 32-bit counter that Postgres compares to decide which row versions a transaction can see.; Sentry 2015: PostgreSQL transaction ID wraparound