# Postmortem: Elevated Errors and Write Unavailability in the Key-Value Store **Incident date:** 2026-07-17 **Duration:** 68 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Storage Platform Team ## Summary On July 17, 2026, the distributed key-value store experienced elevated latency and partial write unavailability in the `us-east` region. A routine rolling deployment restarted several storage nodes while an automated repair process was already moving replicas away from a degraded host group. The combined replica movement exhausted available disk I/O on multiple nodes. Heartbeats and Raft log replication were delayed, causing repeated leader elections across 31% of partitions. Client retries amplified the load, and write success rates fell to 71% at the lowest point. The incident lasted 68 minutes. No acknowledged writes were lost, but some clients received timeout responses for operations that were later committed and therefore may have retried them. Reads using eventual consistency remained mostly available; strongly consistent reads were affected on partitions without a stable leader. ## Customer Impact From 14:07 to 15:15 UTC: - 29% of write requests in `us-east` failed or timed out at the peak. - 11% of strongly consistent reads failed or exceeded the 2-second client timeout. - Eventual-consistency reads had a peak error rate of 1.8%. - Approximately 18% of tenants in `us-east` experienced at least one failed request. - Customers using non-idempotent application-level workflows may have observed duplicate side effects after retrying timed-out writes. - Other regions were not directly affected, although cross-region replication lag reached 14 minutes. No committed data was lost. Offline verification found 2,814 requests that timed out at the client but were subsequently committed. The affected API operations use request identifiers for deduplication, so the store did not create duplicate key versions. ## Detection The incident was first detected by an alert for regional write success rate. Existing alerts for leader-election frequency and repair bandwidth also fired, but they were classified as warnings and did not page the on-call engineer. The deployment system continued the rollout for six minutes after the first symptoms because its health gate used host-level CPU and process availability rather than partition-level leadership and request success metrics. ## Timeline All times are in UTC. - **13:42** — Automated repair begins relocating replicas from a host group after two drives report elevated media errors. - **14:00** — Version `4.18.2` begins rolling deployment to 10% of storage nodes in `us-east`. - **14:04** — The first deployment batch restarts four nodes that are also receiving replacement replicas. - **14:06** — Disk utilization on adjacent nodes exceeds 95%. Raft heartbeat latency begins increasing. - **14:07** — Write latency crosses the service-level objective. Several partitions begin repeated leader elections. - **14:09** — Regional write success alert fires and pages the primary on-call engineer. - **14:11** — The deployment health gate passes because all restarted processes are healthy. The rollout advances to the next batch. - **14:13** — Write success falls below 90%. Client retry traffic reaches 2.3 times the normal request rate. - **14:16** — The on-call engineer pauses the deployment. - **14:20** — The incident is declared SEV-1. Storage, networking, and database reliability engineers join. - **14:25** — Engineers identify correlated disk saturation, replica repair traffic, and Raft election churn. - **14:31** — Automated repair is paused. Disk queues begin to drain, but retry traffic remains elevated. - **14:36** — Client retry limits are reduced at the regional gateway, and exponential backoff is enforced for internal clients. - **14:43** — Stable leaders are restored for most affected partitions. Write success recovers to 94%. - **14:51** — Cross-region replication lag peaks at 14 minutes and begins decreasing. - **14:58** — Write success returns above 99.5%. Strongly consistent read errors return to baseline. - **15:08** — All partitions have stable leaders, and disk utilization falls below 70%. - **15:15** — The incident is resolved after sustained recovery and validation of replica consistency. - **17:40** — Background repair resumes with a manually reduced concurrency limit. ## Root Cause The immediate cause was disk I/O saturation created by two control-plane operations running concurrently: 1. Automated replica repair was copying large replica snapshots away from a degraded host group. 2. A rolling deployment restarted storage nodes, causing additional replica catch-up and Raft log replay. The scheduler treated these operations independently. It limited deployment concurrency by node count and repair concurrency by bytes per second, but it did not account for their combined effect on the same failure domain. Once disk latency increased, Raft heartbeats and log acknowledgements were delayed beyond the election timeout. Followers started elections, leaders stepped down, and partitions temporarily lost stable leadership. Each election generated additional metadata writes and log synchronization, further increasing disk load. Client libraries retried timed-out requests with insufficient jitter and no regional retry budget. The resulting traffic amplification prolonged the incident after the original deployment had been paused. ## Contributing Factors - The deployment scheduler was unaware of active replica repairs. - Repair throttling used throughput rather than disk queue depth or latency. - Several nodes held more replica data than expected because rebalancing had been deferred during the previous capacity expansion. - Raft traffic shared the same disk queue as snapshot transfer and compaction. - Election timeouts were tuned for normal operating conditions and did not tolerate sustained storage latency. - The deployment health gate checked node health but not partition leadership or regional error rates. - Warning alerts for election churn and repair saturation did not page. - Client retry behavior amplified traffic by more than two times during partial failure. - The operational runbook did not identify pausing repair as an early mitigation. ## Resolution and Recovery The team paused both the deployment and automated repair, then reduced retry traffic at the regional gateway. With snapshot transfers stopped, disk queues drained and Raft heartbeats stabilized. Partitions elected leaders and resumed processing writes. After service recovery, the team verified Raft log indexes and replica checksums across all affected partitions. No divergence or loss of acknowledged writes was found. Repair was restarted later with reduced concurrency, and the deployment remained paused pending additional safeguards. ## What Went Well - The regional write-success alert detected the incident within two minutes. - Engineers from storage and database reliability joined quickly. - Pausing repair reduced disk pressure without requiring node replacement or regional failover. - Raft consistency guarantees prevented acknowledged data loss. - Request identifiers allowed timed-out writes to be safely deduplicated. - Replica checksum validation completed without finding divergence. ## What Went Poorly - Independent automation systems initiated conflicting operations. - The rollout advanced after customer-visible errors had begun. - Diagnostic dashboards did not initially correlate repair traffic with deployment batches. - Retry amplification obscured the rate of underlying recovery. - The runbook focused on replacing unhealthy nodes, which would have increased replica movement. - Customers did not receive an incident notification until 31 minutes after impact began. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---:| | Block storage-node deployments while replica repair is active in the same failure domain | Storage Control Plane | P0 | 2026-08-07 | | Add partition leadership and regional request success to deployment health gates | Release Engineering | P0 | 2026-08-07 | | Throttle repair using disk latency and queue depth in addition to bandwidth | Storage Platform | P0 | 2026-08-14 | | Add per-region retry budgets, exponential backoff, and jitter to all supported client libraries | Client Platform | P0 | 2026-08-21 | | Page on sustained leader-election churn and repair-induced disk saturation | SRE | P1 | 2026-08-05 | | Separate Raft log I/O from snapshot and compaction workloads where supported | Storage Platform | P1 | 2026-09-18 | | Rebalance replicas to correct skew introduced during capacity expansion | Capacity Engineering | P1 | 2026-08-28 | | Add a simulation covering concurrent rollout, repair, and client retry amplification | Reliability Engineering | P1 | 2026-09-04 | | Update the incident runbook with repair throttling and retry mitigation procedures | Storage SRE | P1 | 2026-08-04 | | Add customer alerts for regional write degradation within 15 minutes of SEV-1 declaration | Incident Management | P2 | 2026-08-21 | | Review election-timeout tuning under constrained disk conditions | Database Reliability | P2 | 2026-09-11 | ## Lessons Learned Automation that is safe in isolation may be unsafe when combined with other control-plane activity. Concurrency limits must account for shared resources and failure domains rather than only the operation that owns them. Health gates must also measure customer-facing behavior. Process availability did not reflect the state of partition leadership, and allowing the deployment to advance materially increased the scope of the incident. Finally, retry behavior is part of system capacity planning. Clients must treat retries as a limited resource and avoid turning partial degradation into sustained overload.