[09:12:04] Customer: Hi, we’re seeing intermittent write timeouts on our production cluster. Reads are mostly fine, but PUT requests are taking 3–5 seconds and sometimes returning 503.

[09:12:22] Support: Hi Maya. I can help investigate. When did this start, and were there any recent configuration or topology changes?

[09:12:48] Customer: It started around 08:40 UTC, shortly after we added two nodes. The cluster now has 11 nodes across three availability zones.

[09:13:10] Support: Thanks. What consistency level are you using for writes, and what replication factor is configured?

[09:13:24] Customer: Replication factor 3, write quorum 2. Client timeout is 5 seconds.

[09:13:51] Support: Understood. Are the new nodes still receiving partitions? Rebalancing can temporarily increase disk and network pressure.

[09:14:06] Customer: The admin console says rebalancing is 78% complete. Disk utilization on the new nodes is low, but several existing nodes are near 95% I/O utilization.

[09:14:32] Support: That likely explains the latency. During partition migration, the existing owners must serve client traffic while streaming data to the new replicas. If those disks are saturated, quorum acknowledgments can exceed the client timeout.

[09:14:55] Customer: Should we stop the rebalance? We don’t want to risk data loss.

[09:15:14] Support: Don’t force-stop the nodes or remove them from membership. If your platform supports pausing partition transfers, pausing is safe; committed replicas remain in place. Otherwise, reduce the migration rate rather than terminating the process.

[09:15:39] Customer: We have a setting called `rebalance.max_transfers_per_node`. It’s currently 8.

[09:15:52] Support: Lower it to 2 on the saturated source nodes, then watch write latency, disk queue depth, and replica health for about five minutes. That change affects transfer concurrency, not replication durability.

[09:16:21] Customer: Applied. The transfer rate dropped from 410 MB/s to roughly 120 MB/s.

[09:17:03] Support: Good. Are write latency and timeout rates responding yet?

[09:17:18] Customer: P99 PUT latency is down from 4.8 seconds to 1.6 seconds. We’re still seeing a few 503s.

[09:17:41] Support: Please check whether the remaining errors are concentrated on specific partitions or nodes. Also confirm there are no under-replicated partitions.

[09:18:09] Customer: No under-replicated partitions. Most errors point to node `kv-prod-07`, which is also reporting frequent compactions.

[09:18:31] Support: That node may be experiencing compaction amplification on top of the rebalance workload. What are its free disk space and pending-compaction values?

[09:18:46] Customer: 14% free space and 37 pending compactions. Other nodes have between 2 and 6.

[09:19:12] Support: That’s significant. Keep transfer concurrency low and temporarily cap transfers from `kv-prod-07` at 1. Avoid triggering a manual full compaction; it would add more I/O pressure.

[09:19:34] Customer: Done. Would increasing the client timeout help?

[09:19:48] Support: It can reduce visible failures, but it won’t address the underlying saturation. If your application can tolerate it, increasing the timeout from 5 to 8 seconds is a reasonable temporary safeguard. Keep bounded retries with exponential backoff and jitter to avoid a retry storm.

[09:20:15] Customer: We use two retries at 100 ms and 200 ms, without jitter.

[09:20:29] Support: I recommend adding jitter and increasing the delay, for example 250–500 ms before the first retry and 500–1,000 ms before the second. Retry only timeout and unavailable responses, and reuse the same idempotency token for each logical write.

[09:21:02] Customer: We’ve changed the timeout to 8 seconds and enabled jitter. Error rate is now close to zero, and P99 latency is 900 ms.

[09:21:18] Support: Excellent. Let the rebalance continue at the reduced rate. Once it finishes and pending compactions on `kv-prod-07` return to normal, you can gradually restore transfer concurrency while monitoring disk queue depth.

[09:21:39] Customer: Is there anything we should change before adding nodes next time?

[09:21:53] Support: Start with lower transfer concurrency, add nodes during a lower-traffic window, and maintain at least 20–25% free disk space before rebalancing. Alerting on pending compactions and disk queue depth will also provide earlier warning than capacity percentage alone.

[09:22:17] Customer: Got it. Current rebalance progress is 84%, all replicas are healthy, and writes look stable.

[09:22:29] Support: Great. I’ll keep the case open for monitoring. If latency rises again, send the partition-transfer metrics and logs from `kv-prod-07`, and we’ll review them.

[09:22:42] Customer: Will do. Thanks for the help.