[2026-07-11 09:12:04 EDT] Customer - Priya N.: Hi, we’re seeing elevated read latency on our distributed key-value store cluster, mostly for keys in the payments namespace.

[2026-07-11 09:12:31 EDT] Support - Daniel R.: Thanks, Priya. I’m looking at the cluster now. Is this for prod-us-east-kv-03?

[2026-07-11 09:12:46 EDT] Customer - Priya N.: Yes. Five-node cluster, RF=3, reads and writes at QUORUM.

[2026-07-11 09:13:18 EDT] Support - Daniel R.: I see p99 reads climbed from 28 ms to about 410 ms starting at 08:47 EDT. Writes look steadier, around 35 ms p99. Did anything change around then?

[2026-07-11 09:13:55 EDT] Customer - Priya N.: We replaced node kv-03b last night after disk errors. It rejoined around 08:30.

[2026-07-11 09:14:22 EDT] Support - Daniel R.: That lines up. kv-03b is still streaming 1.8 TB of data and has a large read-repair queue. Hot partitions in payments are being routed to it during quorum reads.

[2026-07-11 09:15:03 EDT] Customer - Priya N.: Can we take it out of the read path without losing redundancy?

[2026-07-11 09:15:41 EDT] Support - Daniel R.: Yes. Since RF=3 and the other replicas are healthy, we can temporarily mark kv-03b as joining-only so it continues streaming but won’t serve foreground reads.

[2026-07-11 09:16:10 EDT] Customer - Priya N.: Any risk to writes?

[2026-07-11 09:16:36 EDT] Support - Daniel R.: Low. QUORUM writes will still require two acknowledgments. kv-03b may receive hinted writes during the transition, but the coordinator should satisfy quorum from the other replicas.

[2026-07-11 09:17:08 EDT] Customer - Priya N.: Please go ahead.

[2026-07-11 09:17:44 EDT] Support - Daniel R.: Applied joining-only mode to kv-03b. Watching read latency now.

[2026-07-11 09:18:29 EDT] Customer - Priya N.: Our dashboard already shows p99 down to 96 ms.

[2026-07-11 09:19:02 EDT] Support - Daniel R.: Confirmed from our side. p99 is now 88 ms and falling. There’s still some pressure from compaction on kv-03d.

[2026-07-11 09:19:41 EDT] Customer - Priya N.: Should we throttle compaction?

[2026-07-11 09:20:17 EDT] Support - Daniel R.: I recommend lowering compaction throughput from 160 MB/s to 80 MB/s for the next hour. It will slow cleanup slightly but reduce disk contention.

[2026-07-11 09:20:39 EDT] Customer - Priya N.: Approved.

[2026-07-11 09:21:13 EDT] Support - Daniel R.: Compaction throttle updated. Current read p99 is 44 ms, write p99 is 38 ms.

[2026-07-11 09:22:08 EDT] Customer - Priya N.: That’s back within our SLO. What caused the hot keys to hit the recovering node?

[2026-07-11 09:22:51 EDT] Support - Daniel R.: The token ranges for payments include kv-03b as a natural replica. Once it rejoined, coordinators considered it available before streaming had caught up, so quorum reads occasionally waited on a slow replica.

[2026-07-11 09:23:26 EDT] Customer - Priya N.: Is there a setting to prevent that automatically?

[2026-07-11 09:24:03 EDT] Support - Daniel R.: Yes. Enable defer_reads_until_bootstrapped=true on replacement nodes. That keeps them out of foreground reads until streaming and anti-entropy checks complete.

[2026-07-11 09:24:37 EDT] Customer - Priya N.: Can you add that recommendation to the ticket?

[2026-07-11 09:25:01 EDT] Support - Daniel R.: Added. I also recommend scheduling a repair for the payments namespace after kv-03b finishes streaming.

[2026-07-11 09:25:34 EDT] Customer - Priya N.: ETA on streaming completion?

[2026-07-11 09:26:02 EDT] Support - Daniel R.: At the current rate, about 3 hours 20 minutes. I’ll keep kv-03b in joining-only mode until then unless you ask us to revert sooner.

[2026-07-11 09:26:29 EDT] Customer - Priya N.: Please keep it that way and notify us when it’s safe to restore normal reads.

[2026-07-11 09:27:05 EDT] Support - Daniel R.: Will do. Current cluster status is healthy: all five nodes up, quorum available in every range, no dropped mutations in the last 10 minutes.

[2026-07-11 09:27:42 EDT] Customer - Priya N.: Thanks. We’ll monitor from our side too.

[2026-07-11 09:28:10 EDT] Support - Daniel R.: You’re welcome. I’ll update the ticket again after streaming completes or sooner if latency changes.