Replication: Copies That Keep You Running
Replication means keeping the same data on multiple machines. Done right, it gives you high availability when a node dies, read throughput that scales with demand, durability against disk failure, and the ability to serve users from a nearby data center. Done carelessly, it gives you stale reads, split-brain disasters, and consistency bugs that are nearly impossible to debug in production.
Why Replicate at All?
A single database node is a single point of failure. Replication solves four distinct problems at once.
High Availability
If the primary crashes, a replica can be promoted. The system keeps serving requests instead of going dark.
Read Scaling
Route read queries to replicas. Ten replicas means roughly ten times the read throughput — without touching the primary.
Durability
Data written to three nodes survives two simultaneous disk failures. Replication is your insurance policy.
Geo-Locality
Place a replica in each region. Users in Tokyo read from Tokyo; users in Frankfurt read from Frankfurt. Latency drops from hundreds of milliseconds to single digits.
Single-Leader Replication
The most common model. One node is the primary (leader); it accepts all writes. Every other node is a replica (follower); it applies the same writes in the same order and serves reads.
Read / Write Split
Write path: Client → Primary → replication log → Replica 1, 2, 3 Read path (scale-out): Client → any Replica (or Primary) Result: Writes: 1 node handles them (bottleneck is here) Reads: N nodes share the load
Sync vs Async Replication
The primary can wait for replicas to confirm before acknowledging the write (synchronous), or it can acknowledge immediately and let replicas catch up later (asynchronous). This is the core durability vs latency trade-off.
Multi-Leader and Leaderless
Single-leader is simple but concentrates writes on one node. Two alternative models trade simplicity for flexibility.
Multi-Leader
Multiple nodes accept writes independently and replicate to each other. Common in multi-datacenter setups where you want each region to accept local writes without a cross-continent round trip.
DC-West (leader) ←→ DC-East (leader)
↓ ↓
replicas replicas
Writes accepted locally in each DC.
Conflict resolution required when both DCs
write the same key concurrently.Leaderless (Dynamo-style)
No designated leader. Clients (or a coordinator) send writes to multiple nodes simultaneously and use a quorum to decide success. Used by Cassandra, DynamoDB, and Riak.
N = total replicas W = write quorum R = read quorum Strong consistency: W + R > N e.g. N=3, W=2, R=2 → always overlap Eventual consistency: W + R ≤ N e.g. N=3, W=1, R=1 → fast, but stale reads possible
Replication Lag and Consistency
Async replication means replicas are always slightly behind. That gap — replication lag — causes subtle bugs that only appear under load.
Reading Your Own Writes
A user updates their profile, then immediately reloads the page. If the read hits a lagging replica, they see the old data — their own write is invisible to them.
User writes → Primary (version 42)
User reads → Replica (still at version 41)
User sees: old data ← confusing!
Fix: route reads-after-writes to the primary,
or track the write timestamp and wait for
the replica to catch up before reading.Monotonic Reads
If a user's requests are load-balanced across replicas, they might read version 42, then version 40 on the next request — time appears to go backward. Monotonic reads guarantee that once you see a value, you never see an older one.
Request 1 → Replica A (version 42) → user sees new comment
Request 2 → Replica B (version 40) → comment disappears!
Fix: pin a user's reads to the same replica,
or use a session token to enforce a minimum version.CAP Theorem Connection
Replication lag is a direct consequence of the CAP theorem. Under a network partition, you must choose: keep serving reads (availability) and risk stale data, or refuse reads until replicas are confirmed consistent (consistency). Single-leader async replication chooses availability; synchronous replication leans toward consistency at the cost of latency. See also the Sharding page, where partitioning and replication interact.
Failover and Split-Brain
When the primary fails, a replica must be promoted. This process — failover — sounds simple but hides serious risks.
Automatic Failover Steps
1. Detect primary failure (heartbeat timeout — but is it dead or just slow?) 2. Elect a new primary (choose the most up-to-date replica) 3. Reconfigure clients (update DNS / load balancer to point at new primary) 4. Old primary rejoins as a replica (discard any writes it accepted but never replicated)
Split-Brain: Two Primaries
If the old primary is merely partitioned (not dead), it may still accept writes while the new primary does too. Both believe they are the leader — a split-brain. Writes diverge and data is corrupted.
Live Replication Simulation
Replication Pitfalls
- →Assuming replicas are always current: async replication means lag is normal. Never read from a replica when you need the latest write.
- →Ignoring read-your-own-writes: users who write and immediately read will see stale data unless you route those reads to the primary or track write versions.
- →Treating failover as instant: automatic failover takes seconds to minutes. Design clients to retry and handle temporary unavailability.
- →Split-brain without fencing: two primaries accepting writes simultaneously will silently corrupt data. Always fence the old primary before promoting a new one.
- →Replicating without monitoring lag: a replica that is hours behind is nearly useless for reads and dangerous for failover. Alert on lag, not just node health.
- →Conflating replication with sharding: replication copies the full dataset for redundancy; sharding splits it for scale. You usually need both, and they interact.
- →Synchronous replication to all replicas: one slow or failed replica blocks every write. Use semi-synchronous (one replica confirms) as a practical middle ground.
Key Takeaways
- →Replication keeps copies of data on multiple nodes for availability, read scale, durability, and geo-locality.
- →Single-leader is the simplest model: one primary accepts writes, replicas serve reads. The read/write split is the main scaling lever.
- →Synchronous replication is durable but slow; asynchronous is fast but risks data loss on primary failure. Semi-sync is the common compromise.
- →Replication lag causes reading-your-own-writes and monotonic-reads violations. Route sensitive reads to the primary or track write versions.
- →Failover is risky: split-brain (two primaries) corrupts data. Use fencing or consensus protocols to prevent it.
- →Multi-leader enables low-latency writes across regions but requires conflict resolution. Leaderless (quorum) systems tune consistency vs availability via W and R.
- →Replication and sharding are complementary: replication copies data for redundancy; sharding partitions it for write scale.
Quick Reference: Replication Models
Single-Leader (Primary/Replica)
Simple, strong consistency on the primary. Best for read-heavy workloads. Write throughput limited to one node. Used by PostgreSQL streaming replication, MySQL binlog replication, MongoDB replica sets.
Multi-Leader
Low-latency writes in multiple datacenters. Requires conflict resolution (last-write-wins, CRDTs, or application logic). Used by CouchDB, some MySQL multi-master setups, and geo-distributed PostgreSQL via BDR.
Leaderless (Quorum / Dynamo-style)
No single point of failure. Tunable consistency via W and R quorums. Best for high-availability, eventually-consistent workloads. Used by Apache Cassandra, Amazon DynamoDB, Riak.
Consensus-Based (Raft / Paxos)
Strong consistency with automatic leader election and no split-brain. Higher write latency (quorum round-trip). Used by etcd, CockroachDB, TiKV, and as the coordination layer inside many distributed databases.