[09:12:04] Customer: Hi, we're seeing intermittent write failures on our production key-value cluster. The client returns `QUORUM_NOT_REACHED`, but all five nodes appear healthy.

[09:12:31] Support: Hello! I can help investigate. When did this begin, and were there any recent topology, networking, or consistency-setting changes?

[09:13:02] Customer: It started around 08:40 UTC after we added two nodes. Reads are mostly fine, but about 3–5% of writes fail. We're using replication factor 3 with write quorum 2.

[09:13:28] Support: Thanks. Are the new nodes fully joined and caught up, or are they still streaming partitions from the existing nodes?

[09:13:49] Customer: The dashboard says they're active, but `kvctl topology status` shows several partitions as `MOVING`.

[09:14:12] Support: That likely explains the failures. During partition movement, writes may be forwarded between the old and new owners. If forwarding exceeds the client timeout, the coordinator can report that quorum wasn't reached even though replicas eventually acknowledge the write.

[09:14:35] Customer: Our write timeout is 800 ms. The p99 latency jumped to about 1.4 seconds after scaling out.

[09:14:48] Support: Understood. Please run `kvctl rebalance status --verbose` and check whether any nodes show transfer retries or unusually high pending-handoff counts.

[09:15:21] Customer: Node `kv-04` has 18,442 pending handoffs and 327 transfer retries. The others are below 500 pending.

[09:15:50] Support: That concentration suggests `kv-04` is receiving partitions faster than it can apply them. Could you check its disk latency and compaction queue?

[09:16:24] Customer: Disk write latency is 72 ms on `kv-04`, compared with 8–12 ms elsewhere. The compaction queue is 14. I just noticed it was provisioned with standard network storage instead of the high-IOPS volume used by the other nodes.

[09:16:47] Support: That would account for both the delayed replica acknowledgements and transfer retries. As an immediate mitigation, increase the client write timeout to 2 seconds and throttle rebalancing to reduce pressure on `kv-04`. Avoid lowering the write quorum, since that would reduce durability.

[09:17:10] Customer: What throttle value do you recommend?

[09:17:25] Support: Start with `kvctl rebalance throttle set --mbps 40 --node kv-04`, then monitor disk latency, pending handoffs, and write-error rate for 10 minutes. The setting is reversible and does not interrupt requests.

[09:18:03] Customer: Applied. Write failures are already down to 0.4%, and `kv-04` disk latency is around 35 ms.

[09:18:20] Support: Good. The remaining failures should decline as the backlog drains. The permanent fix is to replace or migrate `kv-04` to storage matching the rest of the cluster. Keep the rebalance throttle in place until that is complete.

[09:18:42] Customer: Could any of the failed writes have been committed despite the client receiving an error?

[09:18:56] Support: Yes. A timeout or `QUORUM_NOT_REACHED` response is indeterminate from the client's perspective: the write may have reached quorum just after the coordinator stopped waiting. If your application retries, use an idempotency token or compare-and-set operation to avoid unintended duplicate effects.

[09:19:25] Customer: These writes set values by key, so retries should be idempotent. We also attach monotonically increasing versions.

[09:19:39] Support: That is appropriate. Confirm the retry uses the same version and payload. If a newer value already exists, the store should reject the stale retry rather than overwrite it.

[09:20:07] Customer: The error rate is now zero, p99 is 620 ms, and pending handoffs on `kv-04` are dropping steadily.

[09:20:22] Support: Excellent. We can identify the mismatched storage on `kv-04` as the root cause, exposed by the additional load from partition rebalancing. The timeout increase and rebalance throttle are temporary mitigations; matching the node's storage profile is the corrective action.

[09:20:46] Customer: Understood. We've scheduled the volume migration for tonight. Thanks for the help.

[09:20:55] Support: You're welcome. After the migration, verify disk latency, allow the rebalance to finish, restore the normal throttle, and then consider returning the write timeout to its previous value once p99 latency is stable.