Latency, Throughput & Availability

Three metrics that define system performance. Understanding these is essential because every design decision trades off one for another. You'll learn what each really means, why they matter, and how they interact.

METRIC 1

Latency: The Time Cost

Latency is the time it takes for a request to travel from client to server and back. It's the delay users experience. Every millisecond counts—especially in competitive markets.

Why Latency Matters

→Users perceive delays > 100ms as sluggish
→Every 100ms delay costs revenue (Amazon: 1% sales loss per 100ms)
→High latency breaks interactivity (forms, searches, real-time features)
→Network delays compound through multiple hops

Types of Latency

Network Latency

Time for data to travel through the internet. Physics limits this— speed of light is ~300,000 km/s. San Francisco to London: ~80ms minimum.

Processing Latency

Time the server takes to compute a response. Depends on database queries, algorithm complexity, I/O operations.

Queueing Latency

Time requests wait when the server is overloaded. The most unpredictable and often largest component under load.

Tail Latency (p99)

Average latency can lie. If 99% of requests complete in 10ms but 1% take 2 seconds, your average might be 30ms—useless for understanding user experience.

p99 latency = the time at which 99% of requests complete. This is what users in the slowest 1% experience.

METRIC 2

Throughput: The Capacity Measure

Throughput is the number of requests (or data units) a system can handle per unit time. Usually measured in requests-per-second (RPS) or transactions-per-second (TPS).

Real-World Example

Your server can process 1 request every 10ms (0.01 seconds). With 4 cores, you can handle ~400 requests per second.

Throughput = Cores × (1 / Processing Time)
         = 4 × (1 / 0.01s)
         = 400 RPS

Throughput vs. Load

Under Load

Incoming requests: 100 RPS, Server capacity: 150 RPS → System keeps up, requests complete quickly

Overloaded

Incoming requests: 200 RPS, Server capacity: 150 RPS → Queue grows, latency spikes, requests timeout

The Throughput Ceiling

Throughput is limited by bottlenecks. If your database can handle 1000 queries/sec but your server logic calls it 2x per request, you're capped at 500 RPS—no matter how many servers you add.

METRIC 3

Availability: The Reliability Guarantee

Availability is the percentage of time your system is operational and responding to requests. It's often expressed as "nines": 99% (two nines), 99.9% (three nines), etc.

The Math of Availability

99% available:3.65 days/year of downtime

99.9% available:8.77 hours/year of downtime

99.99% available:52.6 minutes/year of downtime

99.999% available:5.26 minutes/year of downtime

What Kills Availability

→Server crashes or bugs (unhandled exceptions, memory leaks)
→Database failures or data corruption
→Network issues (packet loss, cable cuts, DNS outages)
→Dependencies go down (third-party APIs, payment processors)
→Hardware failures (disk crashes, power loss)
→Cascading failures (one service down brings others down)
→Bugs in updates (deploy a broken release → instant downtime)

Improving Availability

Redundancy

Multiple servers, replicated databases. If one fails, others take over.

Health Checks

Detect failures fast. Route around broken servers before users notice.

Graceful Degradation

When something fails, still serve users (reduced features, cached data) rather than returning errors.

Monitoring & Alerts

Know instantly when problems occur. Act fast.

INTERACTION

The Impossible Trinity: Tradeoffs

You can't maximize all three. Every architecture decision prioritizes some at the cost of others.

Low Latency + High Availability

Requires replication and caching everywhere. Expensive. Limits throughput.

High Throughput + Low Latency

Requires massive infrastructure. Hurts availability (complex = more failure points).

High Availability + High Throughput

Distributed systems introduce latency (coordination, consensus, replication lag).

Strategic Choices

Smart systems don't try to be perfect at all three. Instead, they prioritize based on their use case:

→Real-time trading: Obsess over latency. Accept lower availability.
→Video streaming: Optimize throughput. Latency doesn't matter (buffering is fine).
→Banking: Require high availability. Accept higher latency (batch processing).
→Chat apps: Optimize latency and availability. Throughput is secondary.

EXPERIMENT

Interactive Simulation

Adjust the controls below to see how latency, throughput, and availability interact in real time. Watch what happens when you overload the system, introduce failures, or scale horizontally.

SUMMARY

Key Takeaways

→Latency is time delay. Tail latency (p99) matters more than average.
→Throughput is capacity per second. Bottlenecks always exist.
→Availability is uptime percentage. Each 'nine' is exponentially harder to achieve.
→These three metrics always trade off. Know what your system prioritizes.
→Scalability is adding capacity (more servers). It doesn't automatically fix latency.
→Understand the breakdown: network + processing + queueing latency.
→Monitor all three in production. Not just availability, not just latency.