[2026-07-14 09:12:04 EDT] Customer (Maya): Hi, we're seeing elevated p99 latency on our distributed key-value store cluster. Reads are normally under 20 ms, but they're hovering around 450 ms since about 08:40.

[2026-07-14 09:12:31 EDT] Support (Jon): Thanks, Maya. Which cluster and region is affected?

[2026-07-14 09:12:48 EDT] Customer (Maya): prod-kv-east-2, running 9 nodes across 3 AZs. Replication factor is 3, read quorum is LOCAL_QUORUM, write quorum is QUORUM.

[2026-07-14 09:13:22 EDT] Support (Jon): Understood. Are writes also affected, or only reads?

[2026-07-14 09:13:44 EDT] Customer (Maya): Writes are up too, but not as bad. p99 write latency is around 120 ms. No outright failures yet.

[2026-07-14 09:14:26 EDT] Support (Jon): I’m checking node health and coordinator metrics now. Do you know if there was a deploy, schema change, or traffic spike near 08:40?

[2026-07-14 09:15:03 EDT] Customer (Maya): We launched a new recommendation job at 08:35 that does range scans over user session keys. It was supposed to be throttled.

[2026-07-14 09:15:41 EDT] Support (Jon): That may explain it. I’m seeing node kv-east-2c-07 coordinating about 42% of read requests, which suggests a hot token range or client-side imbalance.

[2026-07-14 09:16:18 EDT] Customer (Maya): The job uses prefix reads on keys like session:{tenant_id}:{timestamp}. Could that concentrate traffic?

[2026-07-14 09:16:52 EDT] Support (Jon): Yes. If tenant_id has low cardinality or one tenant dominates, those keys can land in a narrow token range. Range scans also bypass some of the normal point-read caching benefits.

[2026-07-14 09:17:30 EDT] Customer (Maya): What should we do immediately?

[2026-07-14 09:18:05 EDT] Support (Jon): First, pause or throttle the recommendation job. Second, reduce scan page size if it must keep running. Third, confirm clients are not pinning requests to a single coordinator.

[2026-07-14 09:18:39 EDT] Customer (Maya): Pausing the job now.

[2026-07-14 09:19:27 EDT] Customer (Maya): Job is paused. p99 reads are already down to around 180 ms.

[2026-07-14 09:20:11 EDT] Support (Jon): Good. I’m also seeing pending compactions on kv-east-2c-07 and kv-east-2c-08. The scan likely increased disk pressure and cache churn.

[2026-07-14 09:20:45 EDT] Customer (Maya): Should we restart those nodes?

[2026-07-14 09:21:09 EDT] Support (Jon): I would not restart while compaction debt is clearing unless a node is unhealthy. Restarts could cause extra read repair and replica catch-up work.

[2026-07-14 09:21:42 EDT] Customer (Maya): Makes sense. Any config changes we can safely apply?

[2026-07-14 09:22:16 EDT] Support (Jon): For the short term, lower the scan concurrency on that workload and set a per-client rate limit. If your client supports token-aware routing, verify it is enabled for all app instances.

[2026-07-14 09:23:03 EDT] Customer (Maya): Token-aware routing is enabled for the API, but I’m not sure about the batch worker.

[2026-07-14 09:23:36 EDT] Support (Jon): Please check that. The coordinator skew I’m seeing is consistent with either hot keys or non-token-aware batch reads. It may be both.

[2026-07-14 09:24:25 EDT] Customer (Maya): Found it. The batch worker uses an older client config with token-aware routing disabled.

[2026-07-14 09:25:02 EDT] Support (Jon): That is likely a major contributor. Before re-enabling the job, update the worker config, cap scan concurrency, and avoid scanning by raw prefix during peak hours.

[2026-07-14 09:25:41 EDT] Customer (Maya): Is there a better key model for this workload?

[2026-07-14 09:26:18 EDT] Support (Jon): Yes. Add a shard component to the session key, such as session:{tenant_id}:{bucket}:{timestamp}, where bucket is derived from a hash. Then scan buckets in parallel with controlled concurrency.

[2026-07-14 09:27:06 EDT] Customer (Maya): Got it. Current p99 reads are now 38 ms and falling.

[2026-07-14 09:27:44 EDT] Support (Jon): That lines up with the job pause. Compaction backlog is also decreasing. I’d keep the job paused until backlog returns to baseline.

[2026-07-14 09:28:21 EDT] Customer (Maya): What baseline should I wait for?

[2026-07-14 09:28:56 EDT] Support (Jon): Pending compactions below 10 per node, disk read latency under 15 ms p99, and no single coordinator handling more than about 20% of cluster reads.

[2026-07-14 09:29:33 EDT] Customer (Maya): Understood. We’ll patch the worker config and relaunch at 10% concurrency after the backlog clears.

[2026-07-14 09:30:02 EDT] Support (Jon): Good plan. I’ll keep this case open and monitor for the next 30 minutes.

[2026-07-14 09:31:18 EDT] Customer (Maya): Thanks. Reads are back to 22 ms p99 now.

[2026-07-14 09:31:49 EDT] Support (Jon): Confirmed from our side. Current cluster health is green, with residual compaction activity but no replica lag or quorum failures.