[09:12:04] Customer: Hi, we're seeing intermittent timeouts and stale reads from our distributed key-value store after adding three nodes yesterday.

[09:12:19] Support: Thanks for reaching out. Which version are you running, and what consistency settings do you use for reads and writes?

[09:12:47] Customer: Version 4.8.2. Writes use quorum, but reads use one replica. We now have nine nodes across three availability zones.

[09:13:11] Support: Understood. Reading from one replica can return older data during replication lag. Are the timeouts occurring on reads, writes, or both?

[09:13:35] Customer: Mostly writes. The p99 write latency jumped from 80 ms to about 2.5 seconds. A few reads return values from several seconds earlier.

[09:14:02] Support: Could you check node health, replication backlog, and clock synchronization? Also, did the new nodes finish rebalancing?

[09:15:18] Customer: Rebalancing still shows 91% complete. Two new nodes have replication queues around 1.8 million entries. Clocks are synchronized within 12 ms.

[09:15:46] Support: That backlog likely explains both symptoms. Rebalancing is consuming disk and network capacity, delaying replication and quorum acknowledgments. What are disk utilization and I/O latency on those nodes?

[09:16:21] Customer: Disk utilization is 88–94%, and write latency is around 70 ms. Existing nodes are closer to 45% utilization and 8 ms.

[09:16:50] Support: The new nodes appear disk-bound. Please reduce the rebalance throughput limit temporarily and verify that client request traffic has priority over background transfers.

[09:17:06] Customer: The rebalance limit is currently unlimited. What value should we start with?

[09:17:28] Support: Start at roughly 25% of the measured disk throughput per node, then monitor client p99 latency and replication backlog for 10–15 minutes. Increase gradually if latency remains stable.

[09:18:03] Customer: We capped it at 40 MB/s per node. Write p99 is already down to 420 ms, and disk utilization is around 72%.

[09:18:26] Support: Good improvement. For stale reads, temporarily switch critical read paths to quorum consistency. That will increase read latency but prevent a lagging replica from serving an older value.

[09:18:49] Customer: Done for checkout and inventory. Read p99 increased from 25 ms to 61 ms, but we aren't seeing stale values now.

[09:19:10] Support: That tradeoff is expected. Keep the stronger consistency setting until rebalancing completes and the replication queues return to normal.

[09:19:31] Customer: One concern: the rebalance estimate now says six hours. Is it safe to leave it running?

[09:19:52] Support: Yes, provided disk utilization stays below your alert threshold, quorum remains available in each partition, and the backlog continues decreasing. Avoid restarting nodes or changing membership during the rebalance.

[09:20:17] Customer: Backlog is decreasing by about 90,000 entries every five minutes. No unavailable partitions.

[09:20:38] Support: That looks healthy. I recommend alerting if the backlog stops decreasing for 15 minutes, disk utilization exceeds 85%, or write p99 rises above your service objective.

[09:21:04] Customer: Understood. Should we keep single-replica reads for noncritical traffic?

[09:21:23] Support: Yes, if occasional stale data is acceptable there. For critical data, quorum reads or version-aware client logic are safer. After the cluster stabilizes, you can evaluate consistency per workload.

[09:21:45] Customer: Great. Write p99 is now 190 ms, and the error rate has returned to baseline.

[09:22:01] Support: Excellent. The likely root cause was unrestricted rebalancing saturating the new nodes' disks, which increased replication lag and delayed quorum writes.

[09:22:18] Customer: That matches what we're seeing. We'll monitor until rebalancing completes.

[09:22:31] Support: Sounds good. I'll keep the case open for 24 hours. If latency rises again, please send the node metrics and rebalance logs for the affected interval.

[09:22:44] Customer: Will do. Thanks for the help.