# Postmortem: Elevated Write Failures in the Distributed Key-Value Store **Incident date:** 2026-07-17 **Duration:** 74 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Storage, Reliability, and Database Platform teams ## Summary On 2026-07-17, the distributed key-value store experienced elevated write latency and failures in the `us-east` region. A routine configuration rollout caused several storage nodes to restart simultaneously. The resulting replica recovery traffic saturated the network links used for both replication and client requests. As request latency increased, application clients retried timed-out writes without sufficient backoff. This amplified load on the remaining healthy replicas and caused leader elections in 18% of partitions. During the incident, 7.3% of write requests failed and p99 write latency peaked at 28.4 seconds. The service remained durable and strongly consistent. We found no evidence of acknowledged writes being lost or stale reads being served. The incident ended after the configuration rollout was halted, retry traffic was rate-limited, and recovery traffic was capped. ## Customer Impact From 14:07 to 15:21 UTC: - 7.3% of write requests in `us-east` returned timeout or unavailable errors. - p99 write latency increased from 85 milliseconds to 28.4 seconds. - Approximately 31% of customers using the affected region experienced at least one failed request. - Read traffic was less affected, but p99 read latency increased to 2.7 seconds. - Applications with aggressive retry policies experienced longer periods of degradation. - Customers using other regions were not directly affected. No committed data was lost. Some clients may have received a timeout even though their write was committed, requiring idempotency checks before retrying. ## Detection The incident was detected by automated alerts for regional write availability and p99 latency. The first alert fired four minutes after the rollout began. The deployment system showed the rollout as healthy because its checks only verified that restarted nodes rejoined their clusters. It did not evaluate replica recovery volume, partition leadership stability, or regional request latency. ## Timeline All times are in UTC. - **14:03** — The configuration rollout begins on storage nodes in `us-east`. - **14:05** — The rollout controller restarts six nodes across three availability zones. - **14:07** — Replica recovery traffic increases sharply and begins saturating storage network links. - **14:09** — Client write latency exceeds timeout thresholds; retry traffic increases. - **14:11** — The first write-availability alert fires. - **14:14** — The on-call engineer acknowledges the alert and begins investigating. - **14:18** — Multiple partitions begin leader elections after replication heartbeats are delayed. - **14:23** — Write failures reach 5.1%; the incident is declared SEV-1. - **14:27** — The storage and network teams join the incident channel. - **14:31** — Engineers identify correlated node restarts but initially suspect a faulty network switch. - **14:38** — Network telemetry confirms link saturation without underlying hardware failure. - **14:42** — The rollout is paused. Twelve nodes have restarted. - **14:47** — Operators reduce the maximum replica recovery bandwidth from 8 Gbps to 2 Gbps per node. - **14:52** — Edge gateways begin rate-limiting retries that do not use exponential backoff. - **14:58** — Network utilization falls below 80%; leader election frequency begins decreasing. - **15:06** — Write success rate returns above 99%. - **15:14** — All affected partitions have stable leaders and healthy replica sets. - **15:21** — Latency returns to normal levels and the incident is resolved. - **17:40** — Data-integrity verification completes with no evidence of lost committed writes. ## Root Cause The immediate cause was a configuration rollout that restarted too many storage nodes within the same recovery domain. The rollout controller enforced a regional concurrency limit of six nodes, but it did not account for partition placement or shared network capacity. The six restarted nodes held replicas for overlapping partition groups. When they rejoined, each attempted to recover its replication backlog at the configured maximum rate. Replica recovery and client traffic shared the same network interfaces and queueing policy. Recovery traffic consumed most available bandwidth, delaying replication heartbeats and client requests. Delayed heartbeats caused healthy leaders to be considered unavailable, triggering unnecessary elections and temporary write unavailability. Client retries amplified the incident. Several internal clients retried timed-out writes immediately and up to ten times. At peak, retry traffic represented 46% of incoming write requests. ## Contributing Factors - The rollout controller limited node count but was unaware of partition placement, recovery domains, and network topology. - Recovery traffic had no regional bandwidth budget and shared priority with client traffic. - The deployment health check only verified node membership. - Replication heartbeat thresholds were too sensitive to short periods of network queueing. - Some client libraries did not enable exponential backoff and jitter by default. - Dashboards did not clearly distinguish original requests from retries. - The operational runbook focused on node health and did not include recovery-bandwidth saturation. - Capacity testing modeled single-node recovery but not concurrent recovery across availability zones. ## Resolution and Recovery We stopped the rollout, reduced per-node recovery bandwidth, and rate-limited excessive client retries. This restored bandwidth for replication heartbeats and client traffic. Once network utilization decreased, partition leaders stabilized and write availability recovered. Replica recovery continued at the reduced rate for approximately three hours without further customer impact. We then ran consistency checks across all affected partition groups. These checks confirmed that quorum-committed writes were present on the expected replicas. ## What Went Well - Availability and latency alerts detected the issue before customer support escalations. - Cross-functional responders joined within 20 minutes of the first alert. - Recovery bandwidth could be adjusted dynamically without restarting nodes. - Quorum replication preserved durability and consistency throughout the incident. - Existing consistency-check tooling allowed rapid verification after recovery. ## What Went Poorly - The rollout system reported success while regional availability was degrading. - Initial investigation focused on hardware failure, delaying identification of recovery traffic as the cause. - Retry amplification was not visible on the primary service dashboard. - No automated guardrail stopped the rollout after write latency crossed the deployment threshold. - The retry rate limiter had to be enabled manually. ## Action Items | Action | Owner | Priority | Due Date | |---|---|---:|---| | Make rollout concurrency aware of partition placement and recovery domains | Storage Platform | P0 | 2026-08-14 | | Automatically pause rollouts when regional write availability or p99 latency breaches thresholds | Release Engineering | P0 | 2026-08-07 | | Reserve network bandwidth and queue priority for client and replication-heartbeat traffic | Network Platform | P0 | 2026-08-21 | | Add a regional bandwidth budget for replica recovery | Storage Platform | P0 | 2026-08-14 | | Enable exponential backoff, jitter, and bounded retries by default in all supported clients | Client Libraries | P1 | 2026-09-04 | | Add retry-attempt metadata and separate retry traffic in dashboards | Observability | P1 | 2026-08-28 | | Update deployment health checks to include leader stability and recovery backlog | Release Engineering | P1 | 2026-08-21 | | Test concurrent multi-zone recovery during quarterly capacity exercises | Reliability Engineering | P1 | 2026-09-11 | | Update the incident runbook with recovery-saturation diagnostics and mitigations | Storage Operations | P2 | 2026-08-07 | | Review heartbeat thresholds under controlled network contention | Database Engineering | P2 | 2026-09-18 | ## Lessons Learned Node-level safety limits are insufficient for distributed systems when nodes share partitions, recovery workloads, and network capacity. Rollout controls must consider failure domains and the system-wide cost of recovery, not only the number of unavailable nodes. Background traffic also needs explicit resource isolation. Replica recovery is necessary for durability, but allowing it to compete equally with quorum heartbeats and customer traffic can turn a routine restart into a regional outage. Finally, retry behavior must be treated as part of service capacity planning. Retries improved resilience for isolated failures but significantly increased load during this incident. Future client and gateway defaults will enforce bounded retries with exponential backoff and jitter.