[09:14:07] Customer — Maya Chen: Hi, we’re seeing intermittent write failures in our production key-value cluster. About 3–5% of PUT requests are returning `503 quorum_unavailable`.

[09:14:31] Support — Daniel Ortiz: Hi Maya. I can help investigate. How many nodes are in the cluster, and what replication and consistency settings are you using?

[09:15:02] Customer — Maya Chen: Five nodes across three availability zones. Replication factor is 3. Writes use quorum consistency. This started around 08:40 UTC after we replaced one node.

[09:15:44] Support — Daniel Ortiz: Thanks. Is the replacement node fully joined and caught up? Please share the output of `kvctl cluster status` and `kvctl rebalance status`.

[09:17:18] Customer — Maya Chen: Here’s the relevant part:

[09:17:19] Customer — Maya Chen: `node-5  JOINING  zone-c  ownership=0%  health=degraded`
`rebalance: paused`
`reason: insufficient disk space on node-3`

[09:17:52] Support — Daniel Ortiz: That explains the quorum errors. The replacement has not received partitions yet, and the paused rebalance likely leaves some replica sets with only one healthy writable replica. What are the disk utilization and recent logs for node-3?

[09:19:11] Customer — Maya Chen: Node-3 is at 94%. Logs show repeated compaction failures: `no space left on device`. The data volume is 500 GB, with about 28 GB free.

[09:19:47] Support — Daniel Ortiz: Understood. Compaction needs temporary working space, so 28 GB may not be usable headroom. Before resuming the rebalance, either expand node-3’s volume or safely remove obsolete snapshots and expired-data files using the store’s cleanup command. Don’t delete storage files manually.

[09:20:23] Customer — Maya Chen: We can expand the volume. Is it safe to do that while traffic is live?

[09:20:40] Support — Daniel Ortiz: Yes, assuming your cloud provider supports online volume expansion. Expand the underlying volume, then grow the filesystem. Once node-3 reports at least 20% free space, run a compaction and confirm it completes before resuming rebalance.

[09:24:56] Customer — Maya Chen: Volume expanded to 750 GB. Filesystem now shows 36% free. We started `kvctl compact --node node-3`; it’s running.

[09:25:20] Support — Daniel Ortiz: Good. While it runs, please check whether hinted handoff queues are growing on the other nodes. That will tell us how much repair traffic to expect.

[09:27:03] Customer — Maya Chen: Nodes 1, 2, and 4 have 8.2 GB total in pending hints. Node-5 has none.

[09:27:28] Support — Daniel Ortiz: That volume is manageable. After compaction finishes, resume the rebalance with the default throttle. Avoid increasing the transfer rate until client error rates stabilize.

[09:31:42] Customer — Maya Chen: Compaction completed successfully. We resumed rebalance. Node-5 is now `SYNCING`, and ownership is at 11%.

[09:32:05] Support — Daniel Ortiz: Excellent. Are `503` responses decreasing?

[09:33:17] Customer — Maya Chen: Yes. They dropped from roughly 4% to 0.2% and are still trending down. P99 write latency is elevated, though—from 85 ms to 240 ms.

[09:33:45] Support — Daniel Ortiz: The latency increase is expected during replica streaming and hinted-handoff replay. Keep the rebalance throttle at its default value and monitor disk latency. If P99 exceeds your service threshold, you can reduce the throttle by 25% without pausing the operation.

[09:38:29] Customer — Maya Chen: Rebalance is at 47%. No new `503` errors for the last three minutes. P99 is back to 130 ms.

[09:38:53] Support — Daniel Ortiz: That’s a healthy recovery pattern. Once rebalance reaches 100%, run `kvctl repair verify` and confirm all five nodes report `READY`. I’d also recommend setting a disk-space alert at 75% and a critical alert at 85% so compaction has enough headroom.

[09:47:16] Customer — Maya Chen: Rebalance finished. All nodes are `READY`, ownership is balanced, and `repair verify` reports no inconsistent ranges. Error rate is back to baseline.

[09:47:39] Support — Daniel Ortiz: Great. The root cause was node-3 exhausting compaction headroom, which paused the rebalance after node replacement and temporarily reduced write-quorum availability. Expanding the volume, completing compaction, and resuming rebalance restored the replica sets.

[09:48:02] Customer — Maya Chen: Confirmed. We’ll add the disk alerts and update our node-replacement runbook. Thanks for the help.

[09:48:15] Support — Daniel Ortiz: You’re welcome. I’ll mark the incident resolved.