[2026-07-11 09:14:03 EDT] Customer - Priya N.: Hi, we're seeing elevated p99 latency on our KV cluster. Writes are mostly fine, but reads jumped from ~18 ms to 240-400 ms about 30 minutes ago.
[2026-07-11 09:14:28 EDT] Support - Alex R.: Hi Priya. I can help. Which cluster and namespace is affected?
[2026-07-11 09:14:47 EDT] Customer - Priya N.: Cluster prod-use1-kv-03, namespace sessions_v2.
[2026-07-11 09:15:22 EDT] Support - Alex R.: Thanks. I’m checking the cluster metrics now. Are clients reporting timeouts or just higher latency?
[2026-07-11 09:15:40 EDT] Customer - Priya N.: About 2% timeout rate from the API tier. Client deadline is 750 ms.
[2026-07-11 09:16:31 EDT] Support - Alex R.: I see read latency concentrated on shards 118-126, mostly replicas on node kv-03-17. That node has high disk read queue depth and compaction activity.
[2026-07-11 09:17:02 EDT] Customer - Priya N.: We did increase TTL churn yesterday. Could compaction be falling behind?
[2026-07-11 09:17:36 EDT] Support - Alex R.: Likely. Your tombstone ratio on sessions_v2 is at 41%, and kv-03-17 is serving leader reads for several hot shards while compacting older SSTables.
[2026-07-11 09:18:12 EDT] Customer - Priya N.: Should we fail leadership away from that node?
[2026-07-11 09:18:55 EDT] Support - Alex R.: Yes. Since quorum health is green, I recommend transferring shard leadership off kv-03-17 first. That will reduce read pressure without restarting the node.
[2026-07-11 09:19:20 EDT] Customer - Priya N.: Please go ahead if it’s non-disruptive.
[2026-07-11 09:20:08 EDT] Support - Alex R.: Starting controlled leader transfer for shards 118-126. Expected impact is brief latency fluctuation while Raft leadership changes.
[2026-07-11 09:21:44 EDT] Support - Alex R.: Transfers completed. I’m seeing p99 reads drop to 92 ms and timeout rate down to 0.4%.
[2026-07-11 09:22:19 EDT] Customer - Priya N.: Confirmed from our side, much better but still above baseline.
[2026-07-11 09:23:01 EDT] Support - Alex R.: Next step is throttling background compaction less aggressively on kv-03-17 for the affected namespace, then scheduling a full compaction window later.
[2026-07-11 09:23:38 EDT] Customer - Priya N.: Is there risk to increasing compaction throughput during business hours?
[2026-07-11 09:24:17 EDT] Support - Alex R.: Some disk contention risk, but we can cap it at 80 MB/s. Current disk utilization has enough headroom now that leaders moved off the node.
[2026-07-11 09:24:52 EDT] Customer - Priya N.: OK, apply the cap.
[2026-07-11 09:25:30 EDT] Support - Alex R.: Applied compaction throughput cap at 80 MB/s for sessions_v2 on kv-03-17. Monitoring read latency and disk queue depth.
[2026-07-11 09:29:46 EDT] Support - Alex R.: Tombstone ratio is decreasing, and p99 reads are now 34 ms. Timeout rate is back below 0.05%.
[2026-07-11 09:30:11 EDT] Customer - Priya N.: Our dashboards match that. What caused the concentration on one node?
[2026-07-11 09:31:02 EDT] Support - Alex R.: Two factors: TTL churn increased tombstones, and the balancer avoided moving leaders during yesterday’s node replacement. That left several high-read shards led by kv-03-17.
[2026-07-11 09:31:41 EDT] Customer - Priya N.: Can we prevent this from happening again?
[2026-07-11 09:32:20 EDT] Support - Alex R.: I recommend enabling leader rebalance after maintenance events and lowering the compaction trigger for this namespace from 35% to 25% tombstones.
[2026-07-11 09:32:58 EDT] Customer - Priya N.: Please open a follow-up ticket with those recommendations.
[2026-07-11 09:33:31 EDT] Support - Alex R.: Done. Follow-up ticket CS-482917 created. I’ll keep this incident open for another 30 minutes of monitoring.
[2026-07-11 09:48:09 EDT] Support - Alex R.: Update: p99 reads have held between 24-31 ms for 15 minutes. No quorum instability, no new client timeout spike.
[2026-07-11 09:48:34 EDT] Customer - Priya N.: Looks stable here. We can close the active incident.
[2026-07-11 09:49:05 EDT] Support - Alex R.: Closing the active incident as mitigated. Root cause summary and recommendations will be added to CS-482917.