[09:14:03] Customer (Maya): Hi, we’re seeing intermittent write timeouts on our production cluster. Reads still look normal.

[09:14:28] Support (Daniel): Hi Maya. I can help investigate. What version are you running, and when did the timeouts begin?

[09:15:02] Customer (Maya): Version 4.8.2. The alerts started around 08:47 UTC, shortly after we added two nodes in eu-west-1c.

[09:15:39] Support (Daniel): Thanks. Are the timeouts limited to a particular region, keyspace, or consistency level?

[09:16:11] Customer (Maya): Mostly eu-west-1. The affected client uses quorum writes with replication factor 3. Error is: “write quorum not reached before 2000ms deadline.”

[09:16:52] Support (Daniel): Understood. Please check the cluster membership view and confirm whether the new nodes are fully active rather than joining or synchronizing.

[09:18:07] Customer (Maya): One new node is active. The other has been in “joining” for 31 minutes. Its streaming progress is stuck at 73%.

[09:18:42] Support (Daniel): That likely explains the increased tail latency. While ownership is moving, some writes require coordination with replicas that are streaming data. Do you see dropped messages, disk saturation, or network errors on that node?

[09:20:15] Customer (Maya): Disk utilization is 98%, and compaction backlog is 412 GB. No obvious network errors.

[09:20:47] Support (Daniel): The node is probably constrained by compaction and streaming I/O. Did you add it with the default concurrent streaming and compaction settings?

[09:21:04] Customer (Maya): Yes. We didn’t change anything before bootstrapping.

[09:21:38] Support (Daniel): As an immediate mitigation, reduce streaming throughput on the joining node and temporarily raise the client write timeout to 5 seconds. Avoid restarting the node; that may discard progress and extend the rebalance.

[09:22:12] Customer (Maya): Is lowering streaming throughput safe while it’s joining?

[09:22:29] Support (Daniel): Yes. It will make the bootstrap take longer, but it should free disk bandwidth for foreground writes and compaction. Start with 50% of the current limit and watch p99 latency, disk queue depth, and pending compactions.

[09:24:18] Customer (Maya): Applied. Disk utilization is down to 86%, and write timeout rate dropped from 7.4% to 0.8%.

[09:24:45] Support (Daniel): Good improvement. Is streaming progress moving again?

[09:25:09] Customer (Maya): Yes, now at 75%. p99 writes are around 1.6 seconds, down from 4.9 seconds.

[09:25:31] Support (Daniel): Great. Keep the 5-second timeout until the node reaches active status and replica ownership stabilizes. Afterward, return it to 2 seconds gradually and confirm there are no hinted handoff or repair backlogs.

[09:26:10] Customer (Maya): We also see about 18 million pending hints. Is that concerning?

[09:26:36] Support (Daniel): It’s consistent with the earlier timeouts, but it should be monitored. Once the node is active, the hints should drain. If the count remains flat or increases for more than 30 minutes, check delivery failures and available disk space before starting any manual repair.

[09:28:02] Customer (Maya): Understood. The timeout rate is now 0.1%, and progress reached 79%.

[09:28:25] Support (Daniel): That indicates the mitigation is working. I’ll keep the case open while bootstrap completes. Please send another update when the node becomes active or if disk utilization returns above 95%.

[10:11:47] Customer (Maya): Update: the node is active. Pending hints are down to 6.2 million, and we haven’t had a write timeout in 12 minutes.

[10:12:14] Support (Daniel): Excellent. I recommend leaving streaming throttled until the hints and compaction backlog are near baseline, then restoring the original limit in small increments.

[10:12:39] Customer (Maya): Sounds good. Can you summarize the root cause for our incident notes?

[10:13:06] Support (Daniel): The second node’s bootstrap saturated disk I/O while streaming and compaction ran concurrently. That delayed replica acknowledgements, causing quorum writes to exceed the 2-second client deadline. Throttling streaming restored capacity for foreground writes and allowed bootstrap to continue.

[10:13:34] Customer (Maya): Perfect. We’ll update our node-addition runbook to throttle bootstrap traffic and monitor compaction backlog.

[10:13:51] Support (Daniel): Great. I’ll mark the incident resolved and leave the case available for 48 hours in case the backlog stops draining.

[10:14:03] Customer (Maya): Thanks for the help.