System Design
Load Balancing
Distributing work across multiple servers to enable scalability, reliability, and optimal resource utilization.
The Concept
Load balancing distributes incoming traffic and computational load across multiple servers to optimize resource utilization, maximize throughput, minimize response time, and avoid overload on any single server.
Without it, performance degrades
Without load balancing, all requests go to a single server. When traffic spikes, that server becomes overloaded, responses slow down, and users experience timeouts and failures.
Where it's critical
- Web applications with millions of concurrent users
- APIs serving multiple client applications
- Microservices architectures
- Database query distribution
- Real-time streaming platforms
Distribute the work fairly so no single server bears all the burden.
Load Balancing Strategies
Round Robin
Requests are distributed sequentially to each server in rotation. Simple but assumes all servers have equal capacity.
Least Connections
Directs new requests to the server handling the fewest active connections. Better for long-lived connections.
Weighted Round Robin
Assigns different weights to servers based on capacity. More powerful servers get more requests.
IP Hash
Routes requests based on client IP address. Ensures a client always connects to the same server (session persistence).
Random
Randomly selects a server for each request. Stateless and works well when servers are homogeneous.
Key Concepts
Design considerations
- Single point of failure: load balancer itself must be redundant
- State management: servers should be stateless when possible
- Latency overhead: load balancer adds minimal but measurable delay
- Sticky sessions: needed for stateful applications
- Cost: load balancers add infrastructure expense
Good load balancing is invisible to users — requests flow smoothly regardless of server load.
Trade-offs
Benefits
- Improved availability through redundancy
- Better resource utilization across servers
- Increased throughput and response time
- Transparent failover to healthy servers
- Easy horizontal scaling
Challenges
- Adds complexity to system architecture
- Load balancer itself becomes critical infrastructure
- Session state management complexity
- Network latency from extra hop
- Requires monitoring and maintenance
Common Patterns
- HAProxy or NGINX for application load balancing
- Cloud providers (AWS ELB, Google Cloud LB, Azure LB)
- DNS-based load balancing for geographic distribution
- Layer 4 (transport) vs Layer 7 (application) balancing
- Redundant load balancers with failover
Real-World Examples
E-Commerce Platform
During sales, load balancer distributes traffic across dozens of application servers to handle peak load.
Social Media Feed
Load balancer routes millions of concurrent feed requests across geographically distributed servers.
Microservices API Gateway
Incoming API requests are load balanced across multiple gateway instances and then to backend services.
Experiment
Interactive Simulation
Interactive Simulation
Load Balancing Live Lab
Adjust traffic, server count, and balancing strategy to see how request distribution, throughput, and queue pressure change in real time.
Requests / tick
24
Throughput
0
Success rate
100.0%
Current utilization
0%
S1
weight 1 • online
0 assigned
0 queued
S2
weight 2 • online
0 assigned
0 queued
S3
weight 2 • online
0 assigned
0 queued
S4
weight 3 • online
0 assigned
0 queued
Average latency
0 ms
Queue pressure and server capacity drive delays.
Queue backlog
0
Requests waiting because servers are at capacity.
Total dropped
0
Requests dropped when no healthy server could accept them.