Case study
A distributed caching layer designed for 300K+ TPS
A self-service caching layer in front of a critical, read-heavy service: same API for clients, adopted team by team, correct under expiry storms and upstream failures.
Project descriptions are intentionally high-level to respect employer confidentiality.
Results
- TPS peak traffic the layer was designed for
- 300K+
- cache hit rate in load tests
- 99.5%
- P99 latency in load tests
- <10 ms
Hit rate and latency were measured in load tests; the 300K+ TPS figure is the peak the layer was designed for.
The problem
- Read peaks of hundreds of thousands of requests per second, almost all reads.
- Clients must adopt it without changing their code.
- Stale data is only acceptable for a short time, and the cache must keep working when the upstream fails.
Decisions and trade-offs
Consistent hashing
Requests for the same key land on the same cache node, so the hit rate holds as the fleet grows instead of every node caching its own copy. Losing a node only moves its share of keys.
Stampede prevention
When a popular entry expires, one request refreshes it while the others wait for that result, so an expiry never turns into a burst of upstream traffic.
Stale-while-revalidate
Serve the cached value immediately and refresh it in the background. When the upstream struggles, briefly serving slightly stale data beats failing: a deliberate trade of freshness for availability.
Invalidation, not just expiry
Time-based expiry alone keeps serving old data after a change, so updates explicitly invalidate cached entries. The trade-off is freshness against upstream load.
Design the cache key
A key that includes more than the identity of the data fragments the cache into many copies. Narrowing it to what identifies the data, while authorization is still enforced on every request, raised reuse without weakening access boundaries.
A reusable layer, not a new system
Packaged as reusable infrastructure as code that teams adopt on their own, instead of a shared system that someone has to operate.