[09:12:04] Customer: Hi, we’re seeing intermittent write failures on our production key-value cluster. The client returns `QUORUM_UNAVAILABLE` for about 2–3% of requests.

[09:12:31] Support: I’m sorry you’re dealing with that. Can you share the cluster topology, replication factor, and consistency level used for writes?

[09:13:02] Customer: Six nodes across three availability zones, replication factor 3, write consistency `QUORUM`. Version 4.8.2. This started after node `kv-prod-05` restarted around 08:40 UTC.

[09:13:40] Support: Thanks. With RF=3 and `QUORUM`, each write needs acknowledgments from two replicas. Is `kv-prod-05` healthy now, and are any other nodes reporting elevated latency or dropped peer connections?

[09:14:11] Customer: `kv-prod-05` shows healthy, but `kv-prod-02` has a lot of messages like `peer request timed out after 2000ms`. CPU is around 45%, memory 72%.

[09:14:47] Support: That suggests the coordinator can reach the replicas, but not within the timeout. Could you check disk latency and compaction activity on `kv-prod-02`, especially p95/p99 write latency and pending compaction bytes?

[09:16:03] Customer: Disk write latency is normally 4–6 ms. It’s currently 180 ms p95 and 1.3 seconds p99. Pending compaction is 640 GB.

[09:16:28] Support: That backlog is likely causing the quorum failures. The restarted node may have triggered replica repair or hinted-handoff replay, increasing disk pressure on `kv-prod-02`. Are streaming or handoff operations active?

[09:17:10] Customer: Yes, `kv-prod-05` is replaying about 210 GB of hints to `kv-prod-02`. We also have a scheduled repair running.

[09:17:34] Support: Understood. I recommend pausing the scheduled repair first. Then throttle hinted-handoff replay to reduce disk saturation. Avoid restarting `kv-prod-02`; that would reduce replica availability further and could increase recovery work.

[09:18:05] Customer: What throttle would you suggest? Current handoff rate is 120 MB/s.

[09:18:34] Support: Start at 30 MB/s and monitor disk p99 latency, pending requests, and quorum errors for 10 minutes. If p99 remains above 500 ms, reduce it to 15 MB/s. The hints will take longer to drain, but foreground traffic should stabilize.

[09:19:22] Customer: Repair is paused and handoff is capped at 30 MB/s. Quorum errors are already dropping.

[09:20:01] Support: Good sign. Please also confirm your client retries use exponential backoff with jitter. Immediate retries can amplify load when replicas are slow.

[09:20:27] Customer: We retry twice with fixed 50 ms delays.

[09:20:51] Support: I’d change that to exponential backoff with jitter, for example approximately 100 ms and 250 ms, with randomized delays. Keep retries bounded so they don’t extend request latency indefinitely.

[09:22:16] Customer: After five minutes, disk p99 is down to 320 ms. No `QUORUM_UNAVAILABLE` errors in the last two minutes. Pending compaction is 590 GB.

[09:22:38] Support: Excellent. Keep the repair paused until handoff finishes and disk p99 returns near baseline. Once stable, resume repair at a reduced throughput and watch for renewed latency.

[09:23:04] Customer: Is there any risk of data loss while hints are draining?

[09:23:30] Support: Not from the backlog alone. Successful `QUORUM` writes were acknowledged by at least two replicas, and hints retain updates for temporarily unavailable replicas. The main risks are exceeding the configured hint-retention window or losing another replica before synchronization completes.

[09:24:02] Customer: Hint retention is 24 hours, and the oldest hint is 48 minutes old.

[09:24:17] Support: That gives you a comfortable recovery window. Keep all nodes online, monitor the oldest-hint age, and alert if it approaches retention. After the backlog clears, run a targeted repair for the affected token ranges.

[09:25:41] Customer: Understood. Disk p99 is now 140 ms and error rate is still zero. We’ll leave the throttle in place and update the retry policy today.

[09:26:03] Support: Great. The incident appears mitigated. Root cause is concurrent hinted-handoff replay and scheduled repair saturating disk on `kv-prod-02`, preventing timely quorum acknowledgments. I’ll keep the case open pending confirmation that the handoff backlog clears and the targeted repair completes.

[09:26:19] Customer: Sounds good. Thanks for the help.