On 19 July 2024, CrowdStrike pushed a routine content update to its Falcon security sensor. It contained a fault that caused Windows machines to crash on boot. Microsoft later estimated that around 8.5 million devices were affected. Airlines grounded flights, hospitals postponed appointments, and payment terminals went dark. Many machines had to be fixed by hand, one at a time.

It's tempting to treat this as something that only happens to giant companies. It isn't. The underlying mistake — shipping a change to everyone at once, with no easy way back — is one that small teams make every week. The consequences are just smaller.

Don't ship to everyone at once

The single most useful habit is a staged rollout. Release to a small slice first, watch, then widen:

  • Deploy to staging and run your smoke tests.
  • Release to a canary — one server, one pod, or a small percentage of users.
  • Watch error rates, latency and business metrics for a set period.
  • Only then roll out to everyone.

On Kubernetes, this can be as simple as a rolling update with readiness probes that actually check something meaningful, so a broken pod never receives traffic. Feature flags let you go further, switching new behaviour on for a few customers before everyone.

Make rollback a button, not a project

When something goes wrong, the question is how fast you can get back to the last good state. If the answer involves rebuilding, SSH sessions or someone who's on holiday, it's too slow.

Keep previous container images tagged and available. Make database migrations backwards-compatible, so the old code can still run against the new schema. Practise rolling back on a quiet afternoon, not for the first time during an incident.

Config is code too

The CrowdStrike fault was in a content file, not in the main program. Many outages come from configuration changes — a firewall rule, a DNS record, a feature flag — that skip the review and testing applied to code. Put configuration in version control, review it, and roll it out with the same care.

Have a plan for when it happens anyway

Things will still go wrong. When they do, a short written runbook — who decides, how to roll back, how to tell customers — turns panic into a checklist. A simple status page and a pre-written customer message save precious minutes.

The lesson

Speed and safety aren't opposites. Teams that deploy small changes often, behind canaries and with fast rollback, end up both faster and more reliable than teams that release rarely and carefully. The goal isn't to never ship a bad change. It's to make sure a bad change only ever reaches a few people, for a few minutes.