Case study
Scaling a time-critical event pipeline 6x
From 200K to 1.2M events per minute on a queue-based pipeline where a late event counts as a failed one.
Project descriptions are intentionally high-level to respect employer confidentiality.
Results
- throughput: 200K → 1.2M events per minute
- 6x
The problem
- Traffic was growing faster than the pipeline could drain it.
- Throughput metrics looked healthy while some events arrived too late to be useful.
- The queue delivers at least once, so duplicates are normal and must be harmless.
Decisions and trade-offs
Isolate the critical path
Different kinds of work get their own queues and consumers, so each one scales and fails independently. A backlog in secondary work never delays a time-critical event.
Scale on pending work, not CPU
Consumers scale on backlog rather than CPU. In a pipeline that mostly waits on I/O, CPU can stay low while work piles up, so it is the wrong signal.
Find the real bottleneck
Throughput plateaued while CPU looked healthy. Investigating under load showed the consumers were bound by a system resource limit, not compute. Removing that constraint lifted the ceiling; adding more instances alone would not have.
Make the effect exactly-once
Delivery is at-least-once, so processing is idempotent: duplicates are detected, and writes are designed so that applying an event twice has the same effect as applying it once.
Alarm on age, not throughput
Healthy throughput can hide a growing backlog. For time-critical work, how long events wait matters more than how many are processed per second, so backlog age and latency define health.