“A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.”

— Leslie Lamport

Everyone knows the quote. It is funny because it is true.

But what the quote does not capture is this: distributed systems rarely fail in the clean, well-modeled way you designed for.

You architect for a node going down.
You get Byzantine clock drift that makes leader election silently split.

You plan for network partitions.
You get a cascading retry storm that takes out your entire control plane in 90 seconds.

After building and breaking distributed systems across cloud providers for the better part of a decade, I have developed a strong opinion:

Most distributed systems do not fail because of novel, exotic bugs. They fail because we systematically underestimate how many things can go wrong at the same time.

This is how distributed systems actually fail — not in theory, but in production.

The Fallacies Are Still Falling

Peter Deutsch’s eight fallacies of distributed computing were written in 1994. They remain one of the most reliable predictors of production incidents:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology does not change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Every incident below traces back to one or more of these assumptions.

We know these fallacies intellectually.
We still build systems as if they are not true in practice.

Failure Mode #1: Cascading Failures

Cascading failure is the big one.

It is the failure mode that keeps SREs up at night because it can turn a small, contained degradation into a full outage faster than any human can respond.

How It Happens

A single upstream service starts responding slowly.

Maybe it is a GC pause.
Maybe it is a disk I/O hiccup.
Maybe one dependency is degraded and nobody noticed yet.

Clients waiting for responses hold open connections. Connection pools exhaust. Thread pools fill. Health check endpoints, often sharing the same application execution path, start timing out.

The load balancer marks instances unhealthy.

Traffic shifts to the remaining instances. Those instances now carry more load. They fail faster.

You have entered the death spiral.

Real-World Case: AWS Kinesis, November 2020

On November 25, 2020, AWS experienced a major outage in us-east-1 that started with Amazon Kinesis.

The root cause was technically a thread-pool exhaustion issue, but the cascade was the real story.

A capacity-scaling operation triggered a race condition in the Kinesis control plane. Some front-end servers consumed excessive memory and became unresponsive. The control plane was not designed to gracefully handle this kind of partial front-end failure, so internal operations began timing out.

Other AWS services that depended on Kinesis — including CloudWatch, which operators relied on to observe the outage — began failing too.

The incident lasted more than 20 hours.

Not because the initial bug was unusually complex, but because the cascading failure made the system hard to observe and hard to recover through normal operations. AWS engineers had to rebuild control-plane capacity manually.

What Actually Helps

Bulkheading

Separate execution resources per dependency.

A slow downstream service should not be allowed to consume all your threads, all your connections, or all your request workers.

Circuit Breakers

Circuit breakers are not just for protecting your service from a slow dependency.

They also protect the dependency from your retry traffic when it is already struggling.

Timeout Budgeting

If your edge SLA is 500 ms, you do not get to set a 5-second timeout to a downstream service.

Work backwards from the user-visible SLA and assign budgets across the call graph.

Graceful Degradation

Your health check endpoint should not depend on the same saturated execution path as user traffic.

If your health check needs the application thread pool to prove that the application is healthy, it may fail exactly when you need it most.

Failure Mode #2: The Retry Storm

Retries are supposed to make systems more resilient.

Uncoordinated retries do the opposite.

At low failure rates, retries are manageable. If you have a 1% failure rate and each failed request is retried three times, your request volume increases only slightly.

But if a downstream service is partially degraded and your failure rate rises to 50%, aggressive retries can send dramatically more traffic to a system that is already struggling.

Now make it worse: the retried requests time out, and those timeouts trigger retries at the next layer up.

That is how a partial degradation becomes a self-inflicted DDoS.

Real-World Case: Cloudflare API Outage, May 2023

On May 13, 2023, Cloudflare experienced a major API and dashboard outage.

The root cause was a database migration that introduced a query performance regression. But retry behavior turned a slow query into a full outage.

The API backend started experiencing elevated latency. Internal and external clients retried aggressively. The database, already struggling with the bad query, began receiving multiple times its normal query volume.

Connection pools saturated across the fleet. The API became fully unresponsive.

The painful part: Cloudflare’s own internal services were also retrying against the same API.

The system was DDoSing itself.

Retry Amplification

The Retry Budget Concept

A retry budget limits retries to a fixed percentage of total requests.

For example:

• 100 original requests per second.
• Retry budget: 10%.
• Maximum retries allowed: 10 per second.

After that, requests fail fast.

That may sound scary because you are intentionally dropping work. In practice, it is often the difference between partial degradation and total collapse.

Key Rules for Retries

1. Only retry idempotent operations.
   Retrying non-idempotent operations is how you double-charge customers, duplicate work, and corrupt state.

2. Use jitter.
   Synchronized exponential backoff is just a coordinated thundering herd with extra steps.

3. Enforce retry budgets.
   Do this at the client layer and, where possible, at the service mesh or infrastructure layer.

4. Fail fast.
   A fast error is usually better than a slow retry that makes the system worse.

Failure Mode #3: Split Brain and Stale Leadership

Distributed consensus is solved technology in the same way bridge construction is solved technology.

The theory is well understood.

You can still build a bad bridge.

Split brain happens when two nodes both believe they are the leader of a cluster. Both accept writes. The data diverges. When the partition heals, you have to pick a side.

The other side’s data is either lost, manually reconciled, or left as corruption disguised as success.

How It Actually Happens in Cloud Environments

In textbooks, split brain happens during a clean network partition.

In cloud environments, it is often caused by partial failures and timing behavior:

• long GC pauses,
• noisy neighbors,
• packet loss,
• asymmetric reachability,
• load balancer behavior,
• stale leases,
• clock drift,
• slow disks,
• overloaded control planes.

The old leader wakes up believing it is still the leader. The new leader has already started accepting writes.

You now have stale leadership.

Real-World Case: Elasticsearch Split Brain

Elasticsearch has a long history with split-brain failure modes.

Before Elasticsearch 7.0, quorum safety depended heavily on the minimum_master_nodes setting. Operators frequently got it wrong.

A common scenario:

• Cluster has 5 master-eligible nodes.
• Operator sets minimum_master_nodes = 2.
• Correct value should be 3, using (N / 2) + 1.
• A network partition separates 2 nodes from 3 nodes.
• Both sides can form a quorum of 2.
• Both sides elect a master.
• You now have two clusters.

Elasticsearch 7.0 improved this by auto-managing quorum.

The broader lesson applies to any system with leader election:

Quorum configuration is not a tuning knob. It is a correctness requirement.

Get it wrong and your data model becomes undefined.

What Actually Helps

Proven Consensus Implementations

Use Raft, Paxos, Zab, or another proven consensus implementation.

Do not roll your own.

Really.

Fencing Tokens

Every successful leader election should produce a monotonically increasing fencing token.

Every operation performed by a leader should include that token.

Downstream systems must reject operations from stale leaders with old tokens.

Lease-Based Leadership With Clock Uncertainty

If your lease is 10 seconds and your maximum clock uncertainty is 500 ms, the leader must stop acting as leader before the lease boundary.

Not at expiry.
Not after expiry.
Before expiry.

Partition Testing

If you have not partitioned your leader from its followers in a test environment, you do not know what will happen in production.

Chaos testing is not theater when it validates correctness properties.

Failure Mode #4: Silent Data Corruption

This is the scariest failure mode because monitoring often does not catch it.

Dashboards are green.
Latency is fine.
Error rate is zero.
The data is wrong.

Distributed systems make guarantees about consistency, ordering, durability, and visibility. Those guarantees are implemented by software. Software has bugs.

When a consistency bug is triggered, the system may not crash. It may continue operating with inconsistent state, and every subsequent operation may make the corruption worse.

Common Patterns

Silent corruption usually appears through patterns like these:

• read-after-write inconsistency that persists beyond the expected eventual consistency window,
• last-write-wins conflicts where the wrong write wins because of clock skew,
• partial replication where a write succeeds on 2 of 3 replicas while the third serves stale reads indefinitely,
• schema migration races where different nodes observe different schemas at the same time,
• replay bugs that apply events twice,
• deduplication bugs that drop valid writes,
• compaction bugs that erase state assumed to be durable.

Real-World Case: GitLab Database Incident, January 2017

On January 31, 2017, GitLab suffered a major data-loss incident.

The chain of failures was brutal:

1. GitLab’s database was experiencing replication lag.
2. An engineer attempted to fix the lag by deleting replication data.
3. The deletion command was accidentally run against the primary database instead of the secondary.
4. Around 300 GB of production data was deleted.
5. Backup procedures that should have protected against this had silently stopped working months earlier.
6. Monitoring that should have alerted on backup failure had also silently stopped working.

The result: roughly six hours of data for more than 5,000 projects was permanently lost.

The deeper lesson is not “humans make mistakes.” That is obvious.

The deeper lesson is this:

Distributed systems fail catastrophically when the protection mechanisms are assumed to be working but are not continuously verified.

Backups are not backups unless restores are tested.

Monitoring is not monitoring unless the monitoring itself is monitored.

What Actually Helps

Continuous Backup Verification

If you are not regularly restoring from backup in a test environment, you do not have a backup.

You have a hope.

End-to-End Data Integrity Checks

Use checksums and integrity verification at the application layer, not only at the storage layer.

Storage can tell you bytes were persisted.

It cannot always tell you whether the bytes are semantically correct.

Invariant Checking

Run background jobs that verify business and system invariants.

Examples:

• Every order must have exactly one payment.
• Every user session must have a corresponding authentication event.
• Every replicated object must exist in the expected number of replicas.
• Every ledger entry must balance.

When invariants are violated, treat it as corruption until proven otherwise.

Audit Logs With External Durability

If your audit logs live in the same database as your primary data, a corruption event can corrupt the evidence.

Critical audit trails need independent durability.

Failure Mode #5: Resource Exhaustion and the Thundering Herd

Recovery can cause the next outage.

That is the thundering herd problem.

A dependency fails. Clients back off. The dependency recovers. All clients reconnect at once. The dependency is overwhelmed by the recovery traffic and fails again.

This failure mode is common in cloud environments because client fleets are large, homogeneous, and synchronized by default.

Real-World Case: Let’s Encrypt OCSP Outage, August 2023

On August 19, 2023, Let’s Encrypt experienced a significant OCSP outage.

The root cause was a database backend that became overwhelmed, but the recovery behavior was the interesting part.

When OCSP went down, clients could not verify certificate revocation status. Many clients cached the failure and retried aggressively.

When responders started coming back online, they were hit with a wave of queued requests from a large client population. The responders could not handle the load and went back down.

The recovery pattern itself became part of the outage.

What Actually Helps

Jitter on Reconnection

Randomized backoff is not optional.

Without jitter, your entire fleet can wake up at the same time and stampede the recovering dependency.

Gradual Recovery

When bringing a service back online, do not immediately expose it to 100% of traffic.

Ramp traffic gradually.

Connection Pool Limits

Connection pools must be sized against downstream capacity, not local convenience.

If each instance can open 500 connections and you have 100 instances, that is 50,000 possible downstream connections.

The downstream service probably cannot handle that.

Admission Control

If you cannot serve a request well, do not accept it.

A fast 503 at the edge is better than a timeout in the middle of the system.

The Composite Failure Problem

Individual failure modes are manageable.

The real disasters come from combinations.

A GC pause causes a leader election.
The new leader’s connection pool is cold.
Latency rises.
Upstream clients retry.
Retries exhaust connection pools across the fleet.
Health checks fail.
The load balancer removes instances.
Remaining instances receive more traffic.
The monitoring system, which depends on the same infrastructure, goes blind.
Operators cannot see what is happening.
Recovery is delayed by hours.

Each individual failure is something you probably designed for.

But have you designed for all of them happening in sequence, with each one making the next more likely?

That is where production systems break.

What I Have Learned

After years of building, operating, and debugging distributed systems, these are the principles that actually matter.

1. Design for Recovery, Not Just Prevention

You cannot prevent all failures.

Stop trying.

Instead, optimize for fast detection and fast recovery.

Ask hard operational questions:

• Can you detect split brain in under 30 seconds?
• Can you drain a misbehaving instance in under 60 seconds?
• Can you restore from backup in under an hour?
• Can you degrade non-critical features automatically?
• Can operators recover the system if the primary dashboard is unavailable?

If the answer is no, your prevention strategy does not matter enough.

When prevention fails, you will be stuck.

2. Observe the Observability

Your monitoring system is also a distributed system.

It will fail too.

Some of the worst incidents start with an absence of telemetry. No graphs. No logs. No traces. No obvious signal.

If you do not monitor the monitoring system from an independent failure domain, you will eventually debug production blind.

At minimum, observe:

• telemetry ingestion health,
• alert delivery health,
• dashboard availability,
• log pipeline lag,
• metric cardinality explosions,
• tracing sampler behavior,
• independent synthetic checks.

3. Test the Failure Modes, Not the Happy Path

Load testing tells you whether your system works under expected pressure.

Chaos testing tells you whether it fails in expected ways.

The highest-value testing is compositional failure testing:

• What happens when a network partition coincides with a deployment?
• What happens when a database failover happens during peak traffic?
• What happens when a leader election happens while clients are retrying?
• What happens when observability fails during an incident?
• What happens when a backup restore is needed while the primary region is degraded?

The compound scenarios are where the real bugs live.

4. Reduce Coordination

The best distributed system is one where nodes do not need to coordinate often.

Every coordination point is a potential failure point.

This does not mean coordination is always avoidable. It means the architecture should continuously push toward independence:

• partition by ownership,
• prefer local decisions,
• avoid global locks,
• avoid synchronous fan-out where possible,
• isolate failure domains,
• make dependencies optional where the product allows it,
• degrade functionality instead of failing globally.

Coordination should be treated as expensive, fragile, and dangerous.

5. Respect the Math

Quorums exist for a reason.
Fencing tokens exist for a reason.
Jitter exists for a reason.
Timeout budgets exist for a reason.

These are not optional best practices.

They are the mathematical and operational foundations that make distributed systems survivable.

Skip them and the system may appear to work in testing, then fail in production in ways that are almost impossible to reproduce.

Closing Thought

Distributed systems fail because we build them with the assumption that rare combinations of events are impossible.

They are not.

In a system processing millions of operations per day, a one-in-a-million event happens constantly. A one-in-a-billion event happens a few times a year.

You only need one.

The systems that survive are not the ones that prevent every failure.

They are the ones that:

• contain failures,
• detect failures quickly,
• degrade deliberately,
• recover automatically,
• and keep enough observability alive for humans to intervene.

Build for that.

Have a war story about distributed systems failure?

I would love to hear it.

These lessons are best learned from someone else’s outage, not your own.