[09:12:04] Customer: Hi, we’re seeing intermittent timeouts and stale reads from our distributed key-value store after adding three nodes yesterday.

[09:12:31] Support: Thanks for reaching out. Are the timeouts affecting all requests, or only specific operations or key ranges?

[09:13:02] Customer: Mostly reads for keys prefixed with `session:`. Writes seem to succeed, but some clients read an older value for 10–20 seconds afterward.

[09:13:40] Support: Understood. What consistency level are the clients using, and did the replication factor change when you expanded the cluster?

[09:14:15] Customer: Replication factor is still 3. Writes use quorum, but I just noticed reads are configured as `ONE`.

[09:14:52] Support: That can explain the stale reads: a `ONE` read may reach a replica that has not yet received or applied the latest write. The timeouts may be separate. Are the new nodes still receiving data through rebalancing or hinted handoff?

[09:15:21] Customer: The admin dashboard says rebalancing is 87% complete. Two new nodes also show high disk utilization and compaction backlog.

[09:16:03] Support: That likely accounts for both symptoms. Rebalancing, compaction, and foreground traffic are competing for disk I/O. Replica updates can lag, while requests routed to the overloaded nodes may exceed the client timeout.

[09:16:37] Customer: Should we stop the rebalance?

[09:17:10] Support: I would avoid stopping it unless latency is causing a critical outage. Instead, reduce the rebalance throughput, temporarily raise the read timeout, and change critical session reads from `ONE` to quorum. That will increase read latency somewhat but should prevent stale results after acknowledged quorum writes.

[09:17:48] Customer: Is changing reads to quorum safe while the rebalance is running?

[09:18:05] Support: Yes, assuming a majority of replicas remain healthy. Monitor unavailable-read errors because quorum reads require enough responsive replicas. I’d apply the change to a small percentage of traffic first.

[09:19:11] Customer: We moved 10% of session reads to quorum. Stale reads disappeared, but p95 latency increased from 18 ms to 46 ms.

[09:19:37] Support: That increase is expected during rebalancing. What are the current client timeout and retry settings?

[09:20:02] Customer: Timeout is 75 ms with two immediate retries.

[09:20:33] Support: Immediate retries can amplify load on already saturated nodes. During the rebalance, try a 200 ms timeout with one retry using exponential backoff and jitter. Keep the total request deadline bounded so application threads do not accumulate.

[09:21:08] Customer: Applied that to one application instance. Timeout errors dropped significantly.

[09:21:29] Support: Good. Please also verify that all nodes have synchronized clocks and that no node is reporting dropped mutations, pending hints, or replication errors.

[09:22:14] Customer: Clocks are within 20 ms. One of the new nodes has 1.8 million pending compaction tasks and about 430,000 pending hints.

[09:22:51] Support: That node is substantially behind. Lowering rebalance throughput should free disk capacity for compaction and hint replay. Avoid running manual repair until the backlog and rebalance complete, since repair would add more I/O pressure.

[09:23:20] Customer: We reduced rebalance throughput by half. Disk utilization is down from 98% to around 82%, and pending hints are decreasing.

[09:24:01] Support: That is a healthy direction. I recommend keeping session reads at quorum, rolling out the adjusted timeout and retry policy gradually, and watching p95/p99 latency, pending hints, compaction backlog, dropped requests, and replica lag.

[09:24:39] Customer: How long after rebalancing finishes should we wait before restoring the original timeout?

[09:25:06] Support: Wait until rebalancing is complete and the hint and compaction queues have remained near zero for at least 30 minutes. Then lower the timeout in stages—200 ms to 150 ms, then 100 ms—while checking tail latency and error rates at each step.

[09:25:43] Customer: Understood. We’ll keep the quorum reads for session data permanently and phase the timeout back down once the cluster stabilizes.

[09:26:02] Support: That sounds appropriate. Based on the evidence, the expansion triggered disk contention and replica lag; `ONE` reads exposed the lag as stale data, and immediate retries intensified the timeouts. If errors rise again after the backlog clears, send us the per-node latency and replication metrics so we can investigate further.

[09:26:18] Customer: Will do. Thanks for helping us narrow it down.

[09:26:24] Support: You’re welcome. I’ll leave the case open while the rebalance completes.