Caching: The Speed Multiplier
Caching is one of the most powerful tools in system design. A well-placed cache can cut latency by 100x and reduce database load by 99%. But caches introduce complexity: stale data, invalidation challenges, and coordination headaches. Learn when to use them, where to put them, and how to avoid the pitfalls.
Cache Hits and Misses
A cache is fast, temporary storage. When data is requested, the system first checks the cache. If the data is there (cache hit), return it instantly. If not (cache miss), fetch from the slow source (database, API, disk) and store it in the cache for next time.
Request Flow: Cache Miss
1. User requests data 2. Check cache → NOT FOUND (miss) 3. Query database → 500ms 4. Store result in cache 5. Return to user Total time: ~500ms Next request (if cache hit): ~5ms
Cache Hit Rate
The percentage of requests that find data in the cache. Even modest hit rates pay huge dividends:
The Core Question
Is this data accessed frequently? Caching only makes sense for hot data—data accessed many times. If you cache data accessed once every hour, you're wasting memory.
Cache-Aside vs Write-Through
Different caching patterns suit different scenarios. The two most common are cache-aside (lazy loading) and write-through.
Cache-Aside (Lazy Load)
The application checks the cache. On miss, loads from the database and populates the cache.
GET user:123 1. Check Redis → MISS 2. Query PostgreSQL 3. Cache result in Redis 4. Return to client GET user:123 (again) 1. Check Redis → HIT 2. Return immediately
Write-Through
On write, update both the cache and the database. Reads always hit the cache.
UPDATE user:123 1. Write to Redis 2. Write to PostgreSQL 3. Return success (both done) GET user:123 1. Check Redis → HIT (always) 2. Return immediately
Other Patterns
Write-Behind (Write-Back)
Write to cache immediately, asynchronously write to database. Fast writes, but risk of data loss if cache fails before flushing.
Refresh-Ahead
Before data expires, preemptively refresh it in the background. Avoids cold starts but requires knowing access patterns.
Cache Eviction: LRU and Beyond
Caches have limited memory. When full, old data must be evicted to make room for new data. Which data should you remove? The answer depends on your eviction policy.
LRU (Least Recently Used)
Evict the item that hasn't been accessed recently. Most common choice. Works well when access patterns are temporal—recently accessed data is likely accessed again soon.
LFU (Least Frequently Used)
Evict the item accessed least often. Better for workloads where some data is accessed far more than others (e.g., popular posts vs obscure pages).
FIFO (First In, First Out)
Evict the oldest item. Simple but naive—doesn't consider access patterns.
Random
Evict randomly. Sometimes performs surprisingly well and has low overhead.
Real Example: LRU in Action
Cache size: 3 items Access pattern: A, B, C, A, D State after each: [A] → Cache: [A] [A, B] → Cache: [A, B] [A, B, C] → Cache: [A, B, C] [B, C, A] → Cache: [B, C, A] (A moved to end) [C, A, D] → Cache: [C, A, D] (B evicted - least recent) LRU timestamp: A > D > C
Cache Invalidation: The Hard Part
Phil Karlton said: "There are only two hard things in Computer Science: cache invalidation and naming things." He was only half joking.
The Problem
When data in the database changes, the cache becomes stale. Serving stale data can cause inconsistencies, bugs, or data corruption.
Nightmare Scenario:
User updates their address. The database is updated. But their old cached address is served to the shipping service. Package arrives at the wrong place.
Invalidation Strategies
TTL (Time-To-Live)
Expire data automatically after N seconds. Simple but can serve stale data for up to N seconds. Good for data that's okay to be slightly outdated (e.g., tweet counts, article views).
Explicit Invalidation
When data changes, immediately delete it from the cache. Guarantees freshness but requires coordination (delete from cache AND database in sync).
Event-Driven Invalidation
Use message queues or event streams. Database emits "address_updated" event → cache service listens and deletes relevant entries. Decoupled and scalable.
Versioning / Tags
Instead of deleting, create a new version. Use cache tags to group related entries. Invalidate by tag rather than individual keys.
The Trade-off
Fast staleness: TTL is simple but can serve old data.
Immediate freshness: Explicit invalidation is complex and prone to consistency bugs (what if the delete fails?).
In practice: Most systems use a combination. TTL as a safety net, explicit invalidation for critical data.
Cache Layers: Where to Put Caches
Caches can exist at multiple layers. Each layer solves different problems.
Client-Side Cache
Browser caches (localStorage, sessionStorage) or mobile app caches. Eliminates network round-trips entirely.
localStorage.setItem("user", JSON.stringify(user))
// Next time
const cached = JSON.parse(localStorage.getItem("user"))Best for: Static assets, user preferences, session data
Application Cache (In-Memory)
In-process cache (HashMap, LRU cache) within your application. Fast (same process), but not shared across instances.
// In-process HashMap
const cache = new Map()
function getUser(id) {
if (cache.has(id)) return cache.get(id)
const user = db.query(id)
cache.set(id, user)
return user
}Best for: Hot data, expensive computations, within single instance
Distributed Cache (Redis, Memcached)
Separate cache layer shared by all servers. Solves the single-instance problem but adds network latency.
GET user:123 App1: Redis miss → query DB → set Redis App2: Redis hit → return instantly App3: Redis hit → return instantly
Best for: Shared state, multiple instances, high scalability
Database Cache (Query Cache)
The database caches query results and table pages in memory. Automatic, no code needed. But you don't control it.
Best for: Automatic optimization, zero overhead
The Ideal Stack
User browser ↓ (static assets cached) CDN ↓ (edge cache hit?) Client application ↓ (in-process cache hit?) Redis (distributed cache) ↓ (cache miss → hit database) Database (table page cache)
Live Cache Simulation
Cache Pitfalls to Avoid
- →Caching too much: Every cached item requires invalidation logic. Cache only hot data.
- →Forgetting TTL: Data lives forever → stale data forever. Always set expiration.
- →Cache stampede: When a key expires, 1000 requests hit the database simultaneously. Use locks or refresh-ahead.
- →Double-write inconsistency: Update cache and database separately → if one fails, they diverge. Use transactions or queues.
- →Ignoring cache misses: Cold start is slow. Preload critical data on startup.
- →Not measuring hit rate: You can't improve what you don't measure. Monitor cache metrics.
- →Over-indexing on latency: A cache that's complex and buggy is worse than no cache. Start simple.
Key Takeaways
- →Caches are fast temporary storage. A 100x latency improvement is common.
- →Cache-aside (lazy load) is simple but has cold starts. Write-through is warm but slower for writes.
- →LRU is the standard eviction policy. Choose based on your access patterns.
- →Cache invalidation is hard. Use TTL for simplicity, explicit invalidation for correctness, or combine both.
- →Cache at multiple layers: client, application, distributed cache, database.
- →Monitor hit rate. A cache with 50% hit rate is half as effective as one with 90%.
- →Not all data should be cached. Only hot data that's accessed frequently.
- →Caching is a trade-off: speed vs complexity. Start simple, add complexity only when needed.
Quick Reference: When to Use Which
Browser Cache (LocalStorage)
Static assets, user prefs, session data. Don't use for sensitive data.
In-Memory Map (HashMap)
Hot data within a single instance. Simple, fast, but not shared.
Redis / Memcached
Distributed cache shared by all servers. Best for scaling, but adds complexity.
CDN Cache
Edge caching for static content. Geographic distribution, fast globally.