# Postmortem: Elevated Write Failures and Read Latency **Incident date:** 2026-07-08 **Duration:** 78 minutes **Severity:** SEV-1 **Status:** Resolved **Services affected:** Distributed key-value store, metadata API, session service ## Summary On 2026-07-08, the distributed key-value store experienced elevated write failures and read latency in the `us-east` region. A routine deployment introduced a defect in replica health handling that allowed lagging replicas to remain eligible for leadership. During a concurrent network degradation between two availability zones, repeated leader elections selected replicas that had not fully applied recent log entries. The resulting election churn reduced write availability and caused request queues to grow across the cluster. Automated rollback removed the defective version, but recovery was delayed by overloaded nodes rejoining too quickly and triggering additional elections. No acknowledged writes were lost. A small number of eventually consistent reads returned stale values within the configured consistency window. ## Customer Impact From 14:07 to 15:25 UTC: - 18.6% of write requests in `us-east` failed or timed out. - 7.4% of read requests exceeded the 500 ms latency objective. - Read latency peaked at 4.8 seconds at the 99th percentile. - Approximately 22% of active customer tenants experienced at least one failed request. - Three dependent services entered degraded modes because they could not reliably update session and metadata records. - Eventually consistent reads could return values up to 47 seconds stale. - Strongly consistent reads remained correct but experienced elevated latency and intermittent timeouts. Clients with exponential backoff generally recovered without intervention. Clients using immediate retries amplified load and experienced higher failure rates. ## Detection The incident was first detected by an alert on regional write availability at 14:09 UTC, two minutes after customer impact began. A separate leader-election-rate alert fired at 14:11 UTC. The deployment pipeline did not automatically halt because its health gate evaluated aggregate global availability. Healthy traffic in other regions masked the regional failure. ## Timeline All times are in UTC. - **13:42** — Version `v4.18.0` begins deployment to 10% of storage nodes in `us-east`. - **13:51** — Canary metrics remain within thresholds. Deployment proceeds to the remaining nodes. - **14:03** — Deployment completes. - **14:06** — Packet loss increases between availability zones `use1-az2` and `use1-az3`. - **14:07** — Several shard leaders miss heartbeat deadlines. Leader elections begin. - **14:09** — Regional write availability falls below 99.5%; the on-call engineer is paged. - **14:11** — Leader-election-rate and request-queue-depth alerts fire. - **14:14** — On-call confirms failures are concentrated in shards whose leaders moved between availability zones. - **14:18** — Incident declared SEV-1. Storage, networking, and client-platform teams join. - **14:22** — Network telemetry confirms intermittent packet loss, but the observed election rate is higher than expected for the measured degradation. - **14:27** — Engineers correlate the start of abnormal elections with `v4.18.0`. - **14:31** — Rollback to `v4.17.6` begins. - **14:39** — First rolled-back nodes restart and rejoin. Election churn temporarily increases as recovering nodes become eligible for leadership before their logs are fully applied. - **14:44** — Write failures peak at 31.2%. - **14:49** — Operators disable automatic leadership eligibility for rejoining nodes. - **14:53** — Rollback completes on all storage nodes. - **15:01** — Leader-election rate begins declining. - **15:08** — Write success rate recovers above 99%. - **15:17** — Read latency returns below the service-level objective. - **15:25** — All regional health indicators remain stable for ten minutes; incident impact ends. - **16:10** — Consistency audit confirms no acknowledged writes were lost. - **18:30** — Network provider confirms a faulty top-of-rack switch caused the packet loss. ## Root Cause The primary cause was a regression in `v4.18.0` in the logic that determines whether a replica is eligible to participate in leader elections. The release replaced a single replication-lag threshold with separate time-based and log-position-based checks. Due to an inverted Boolean condition, a replica was marked eligible when either check passed rather than only when both checks passed: ```text eligible = lag_time_within_limit || log_position_within_limit ``` The intended condition was: ```text eligible = lag_time_within_limit && log_position_within_limit ``` During normal operation, both checks usually agreed, so the defect did not appear in canary testing. Under partial network loss, some replicas continued receiving periodic heartbeat metadata while falling behind in log replication. Their time-based lag appeared acceptable even though their log position was substantially behind. These replicas repeatedly entered elections, became leaders, and then stepped down when they could not establish a stable quorum or serve writes within the lease deadline. This created election churn, increased cross-zone traffic, and consumed CPU and connection capacity needed for replication and request processing. ## Contributing Factors - The network degradation created the asymmetric failure mode required to expose the defect. - Canary testing did not include packet loss with heartbeat delivery continuing while replication traffic was delayed. - Deployment health gates used global availability rather than per-region and per-shard indicators. - Rejoining nodes became eligible for leadership before completing log replay, prolonging recovery. - Retry traffic from some clients increased request volume by approximately 2.3 times during the incident. - Election-rate alerts identified the symptom but did not include replica lag or recent deployment information in the alert context. - The rollout completed before the cluster encountered enough leader elections to exercise the defective path. ## Resolution and Recovery The incident was mitigated by rolling back `v4.18.0` to `v4.17.6` and temporarily disabling leadership eligibility for nodes until they completed log replay and remained healthy for five minutes. After election churn subsided, request queues drained without manual data repair. Engineers then ran a consistency audit comparing committed log indexes, replica checksums, and sampled key histories across all affected shards. The audit found no lost acknowledged writes or divergent committed state. The faulty network switch was removed from service by the provider later that day. ## What Went Well - Regional availability and election-rate alerts fired within four minutes of impact. - Cross-functional incident response was established within eleven minutes. - The team correlated the regression with the deployment and initiated rollback quickly. - Quorum-based writes prevented acknowledged data loss. - Existing consistency-audit tooling allowed data integrity to be verified the same day. ## What Went Poorly - Global deployment gates failed to stop a region-specific regression. - Rollback initially worsened election churn because restart behavior did not account for replica catch-up. - The relevant election code lacked tests for disagreement between time-based and log-position-based lag signals. - Client retry behavior was not consistently bounded, increasing load during recovery. - Operational dashboards required manual correlation across deployment, election, replication, and network views. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---:| | Correct the replica eligibility condition and add regression tests for mismatched lag signals | Storage Consensus | P0 | 2026-07-18 | | Require replicas to complete log replay and pass a stability window before leadership eligibility | Storage Consensus | P0 | 2026-07-25 | | Add fault-injection tests covering asymmetric packet loss, delayed replication, and successful heartbeats | Reliability Engineering | P0 | 2026-07-31 | | Change deployment gates to evaluate availability and election rates per region and per shard cohort | Release Engineering | P0 | 2026-07-24 | | Automatically pause rollouts when leader-election rate exceeds the regional baseline | Release Engineering | P1 | 2026-08-05 | | Add replica log-position distribution and deployment version to election alerts | Observability | P1 | 2026-07-29 | | Implement adaptive election backoff during repeated leadership failures | Storage Consensus | P1 | 2026-08-14 | | Enforce retry budgets and jittered exponential backoff in supported client libraries | Client Platform | P1 | 2026-08-21 | | Publish retry guidance and contact customers generating unbounded retry traffic | Developer Relations | P2 | 2026-08-28 | | Build a unified incident dashboard for elections, replication lag, request queues, and network health | Observability | P2 | 2026-09-04 | ## Lessons Learned Replica eligibility is a safety-critical decision and must depend on authoritative replication state, not a single freshness proxy. Signals that are strongly correlated during normal operation can diverge under partial failures, and tests must explicitly exercise those disagreements. Recovery behavior also requires the same scrutiny as steady-state behavior. Although rollback removed the defective code, allowing recovering nodes to participate immediately in elections extended the outage. Future changes will treat rejoin and catch-up transitions as explicit deployment and failure-testing scenarios.