From 120 minutes to under 5: real-time event processing at 24× speed

The consumers couldn't keep up with incoming events. Lag built up continuously on the Kafka topics, so features that depended on real-time status showed stale data, and the backlog grew during peak hours.

• ▸Around 100,000 user status-change events per hour from upstream products, with surges at peak hours.
• ▸Events flow through Kafka topics into services that power real-time product features.

As the senior engineer assigned to the problem, I owned the optimization end to end: finding where time went, designing the fix, and rolling it out.

I measured where processing time was spent and looked for gaps in the flow. The biggest cost was on the read path: processing each event needed data that took several API calls to fetch, and the same calls were made over and over.

• ▸Added caching for the reads that previously required multiple API calls, so most lookups were served without a network round trip.
• ▸Optimized the remaining steps of the pipeline end to end once the main bottleneck was gone.
• ▸Added metrics for event volume and processing time so lag and scale were visible instead of discovered through user impact.

Processing 30,000 events went from about 120 minutes to under 5 minutes, a 24× improvement. That throughput is well above the average arrival rate of ~100K events per hour, so the pipeline keeps up with traffic instead of falling behind.

• ▸Design for peak traffic, not the average. A pipeline that only just keeps up on a normal hour will fall behind during every surge.
• ▸Measure before optimizing. The time went into repeated network calls, not compute, so caching those reads did more than any code-level tuning could.
• ▸Instrument event pipelines from day one. Consumer lag, per-stage latency, and throughput metrics surface scale problems before users notice stale data.