# Postmortem: Elevated Write Failures in Distributed Key-Value Store

**Incident date:** 2026-07-24  
**Duration:** 54 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Key-value store write path, replication, management API

## Summary

On July 24, 2026, our distributed key-value store experienced elevated write latency and request failures in three production regions. A routine configuration rollout reduced the time allowed for replica acknowledgements. During an unrelated network degradation between two availability zones, this timeout caused healthy but delayed replicas to be marked unavailable.

The resulting replica reassignments generated additional control-plane traffic and data movement. This feedback loop overloaded cluster coordinators, delayed membership updates, and caused write requests requiring quorum acknowledgement to time out.

We restored service by rolling back the timeout configuration, pausing replica rebalancing, and restarting overloaded coordinators in a controlled sequence. No acknowledged writes were lost. Some clients that did not use idempotency tokens may have created duplicate application-level operations after retrying requests whose responses timed out.

## Customer Impact

From 14:07 to 15:01 UTC:

- 18.6% of write requests returned `503 Service Unavailable` or timed out.
- Successful write latency increased from a baseline p99 of 42 ms to a peak of 8.4 seconds.
- Reads using strong consistency experienced a 6.1% error rate.
- Eventually consistent reads remained available, with p99 latency below 190 ms.
- 27% of production clusters in `us-east-1`, `us-central-1`, and `eu-west-1` were affected.
- Cluster creation, scaling, and configuration changes were delayed by up to 43 minutes.
- No acknowledged data was lost, and post-incident consistency checks found no unrepaired replica divergence.

Customers using exponential backoff and idempotency tokens generally recovered without intervention. Customers with fixed-interval retries generated additional traffic and, in a small number of cases, observed duplicate application-level transactions.

## Detection

The incident was first detected by an automated alert for regional write quorum failures. A customer support ticket arrived two minutes later.

Our initial alerts identified symptoms but did not immediately expose the relationship between replica acknowledgement timeouts, rebalancing activity, and coordinator saturation. This increased diagnosis time.

## Timeline

All times are in UTC.

| Time | Event |
|---|---|
| 13:42 | Deployment begins for a configuration change reducing the replica acknowledgement timeout from 2,000 ms to 500 ms. |
| 13:55 | Rollout reaches 100% of storage nodes. Health checks and synthetic traffic remain normal. |
| 14:03 | A network provider begins maintenance that introduces intermittent packet loss and 300–700 ms latency between two availability zones in `us-east-1`. |
| 14:05 | Storage nodes begin marking delayed replicas as unavailable after acknowledgement deadlines are exceeded. |
| 14:07 | Automated alert fires for elevated write quorum failures in `us-east-1`. Incident declared SEV-1. |
| 14:09 | Replica rebalancing begins across affected partitions. Coordinator CPU usage rises above 90%. |
| 14:11 | First customer support ticket reports write timeouts. |
| 14:14 | The same timeout-driven behavior appears in `us-central-1` and `eu-west-1`, where ordinary latency spikes are sufficient to trigger replica eviction. |
| 14:18 | On-call engineers suspect a network issue and engage the infrastructure networking team. |
| 14:23 | Engineers identify a sharp increase in replica membership changes following the configuration rollout. |
| 14:27 | The team attempts to roll back the timeout configuration. Rollback propagation is delayed because coordinators are saturated. |
| 14:31 | Replica rebalancing is manually paused, reducing control-plane load and cross-zone data transfer. |
| 14:36 | Configuration rollback begins reaching storage nodes. Write success rates start improving. |
| 14:41 | Two overloaded coordinators are restarted sequentially after leadership is transferred to healthy peers. |
| 14:47 | Write error rates fall below 2%; latency remains elevated while request queues drain. |
| 14:53 | Network latency in `us-east-1` returns to normal. |
| 15:01 | Write error rates and latency return to baseline. Incident marked mitigated. |
| 15:24 | Replica rebalancing resumes with a reduced concurrency limit. |
| 17:10 | Consistency scans complete with no data loss or unrepaired divergence detected. |

## Root Cause

The immediate cause was a configuration change that reduced the replica acknowledgement timeout from 2,000 ms to 500 ms.

The new value had been validated under steady-state load and during complete replica failures, but not under partial network degradation. When cross-zone latency exceeded 500 ms, storage nodes treated slow replicas as failed even though those replicas remained healthy and continued processing requests.

Each perceived failure triggered replica reassignment. Replica reassignment required:

1. Publishing a membership update through the cluster coordinator.
2. Selecting a replacement replica.
3. Copying partition data to the replacement.
4. Updating routing metadata across storage and gateway nodes.

As more replicas were classified as unavailable, coordinators processed a rapidly increasing number of membership updates. The associated data movement also consumed network and disk capacity needed by foreground traffic.

Coordinator saturation delayed membership propagation, causing different nodes to temporarily hold inconsistent views of replica availability. Write requests were repeatedly routed to replica sets that could not obtain quorum before their deadlines, producing timeouts and retries. Those retries increased load and reinforced the failure cycle.

## Contributing Factors

- The timeout was deployed globally rather than through a regional canary.
- Pre-production tests covered hard failures but not degraded networks with packet loss and variable latency.
- Replica health used the same timeout value for request deadlines and failure detection, coupling transient slowness to topology changes.
- Rebalancing had no rate limit tied to coordinator load or foreground request latency.
- The rollback mechanism depended on the coordinators that were saturated during the incident.
- Several internal clients retried writes at fixed intervals without jitter, increasing load during recovery.
- Dashboards showed replica failures, coordinator saturation, and rebalancing traffic separately, obscuring their causal relationship.
- The change review classified the timeout adjustment as low risk because it did not modify executable code.

## Resolution and Recovery

We mitigated the incident by:

- Pausing automatic replica rebalancing.
- Restoring the replica acknowledgement timeout to 2,000 ms.
- Transferring leadership away from overloaded coordinators and restarting them sequentially.
- Temporarily reducing background repair and compaction concurrency.
- Applying stricter retry rate limits at regional gateways.
- Resuming rebalancing only after coordinator load and write latency returned to normal.

After recovery, we ran full consistency scans on affected partitions and compared commit indexes across replicas. We found no loss of acknowledged writes and no remaining divergence.

## What Went Well

- Quorum replication prevented acknowledged data loss.
- Automated alerts detected the incident before most customer reports arrived.
- The team paused rebalancing without requiring a full cluster restart.
- Coordinator leadership transfers allowed controlled recovery while preserving availability.
- Existing consistency tooling provided high confidence in data integrity within two hours.

## What Went Poorly

- The deployment process allowed a failure-detection parameter to reach all regions without a canary period.
- Rollback propagation was impaired by the same control-plane saturation caused by the change.
- Initial investigation focused on the network degradation rather than why routine latency triggered widespread topology changes.
- Customer-facing status updates initially described the issue as regional, although two additional regions were already affected.
- Retry guidance for client libraries was inconsistent across supported languages.

## Action Items

| Action | Owner | Priority | Target date |
|---|---|---:|---|
| Separate request acknowledgement deadlines from replica failure-detection thresholds. | Storage Platform | P0 | 2026-08-14 |
| Require a one-region canary and a 30-minute observation period for topology or quorum-related configuration changes. | Release Engineering | P0 | 2026-08-07 |
| Add degraded-network tests covering latency, packet loss, reordering, and partial partitions. | Reliability Engineering | P0 | 2026-08-21 |
| Add coordinator-load and foreground-latency limits to automatic rebalancing. | Control Plane | P0 | 2026-08-21 |
| Move emergency configuration rollback to an out-of-band distribution path independent of cluster coordinators. | Infrastructure | P1 | 2026-09-04 |
| Add a circuit breaker that suppresses replica eviction during correlated cross-zone latency events. | Storage Platform | P1 | 2026-09-11 |
| Create a dashboard correlating replica timeouts, membership changes, coordinator load, and rebalancing throughput. | Observability | P1 | 2026-08-14 |
| Alert when replica membership change rates exceed historical bounds. | Observability | P1 | 2026-08-07 |
| Update all official client libraries to use exponential backoff with jitter and idempotency tokens by default. | Developer Experience | P1 | 2026-09-18 |
| Classify quorum, timeout, and failure-detection configuration changes as high risk in the change-management system. | Production Engineering | P1 | 2026-08-07 |
| Add a runbook for disabling rebalancing and transferring coordinator leadership during control-plane saturation. | Incident Management | P2 | 2026-08-14 |
| Review status-page procedures for detecting and communicating multi-region expansion. | Customer Reliability | P2 | 2026-08-21 |

## Lessons Learned

A slow replica is not necessarily a failed replica. By using one aggressive timeout for both request handling and failure detection, we allowed a transient network condition to trigger expensive topology changes. Those changes consumed the resources required for the system to recover and transformed a limited network degradation into a multi-region outage.

Future changes to quorum behavior, replica health, and topology management will receive the same testing and rollout controls as executable changes to the storage engine.