# Postmortem: Distributed Key-Value Store Write Unavailability

**Incident Date:** 2026-07-09  
**Duration:** 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services Affected:** Core key-value store, session service, feature flag service

## Summary

On July 9, 2026, the distributed key-value store experienced elevated write latency and partial write unavailability in the `us-east-1` region. Reads remained mostly available, but writes requiring quorum failed for approximately 47 minutes.

The outage was triggered by an overloaded compaction process on one storage shard group during a rolling deploy. A configuration change reduced the effective write timeout while compaction increased disk latency. This caused repeated leader stepdowns, quorum write failures, and client retries that further amplified load.

## Customer Impact

From 14:06 to 14:53 UTC:

- Approximately 18% of write requests in `us-east-1` failed or timed out.
- Median write latency increased from 12 ms to 190 ms.
- p99 write latency peaked at 8.4 seconds.
- Some customers experienced failed login sessions, delayed feature flag updates, and stale configuration reads.
- No confirmed data loss occurred.
- Cross-region replication lag peaked at 11 minutes and recovered by 15:21 UTC.

## Timeline

All times are UTC.

| Time | Event |
|---|---|
| 13:42 | Rolling deploy begins for key-value store version `2026.07.09.1`. |
| 14:03 | First shard group in `us-east-1` starts background compaction after restart. |
| 14:06 | Write latency alert fires for shard group `use1-kv-17`. |
| 14:09 | On-call engineer acknowledges alert and begins investigation. |
| 14:13 | Error budget burn alert fires for dependent session service. |
| 14:16 | Metrics show elevated disk await time and leader election churn. |
| 14:21 | Rolling deploy is paused. |
| 14:25 | Team identifies quorum write failures concentrated on three replica sets. |
| 14:31 | Write timeout configuration change is identified as part of the deploy. |
| 14:35 | Configuration rollback begins. |
| 14:41 | Leader stability improves, but client retry traffic remains elevated. |
| 14:46 | Rate limits are temporarily applied to non-critical bulk write clients. |
| 14:53 | Write success rate returns to normal. Incident marked mitigated. |
| 15:21 | Replication lag returns to baseline. |
| 16:10 | Full service validation completed. |

## Root Cause

The root cause was an unsafe interaction between disk-heavy compaction, leader election behavior, and a reduced write timeout introduced in the deploy.

Version `2026.07.09.1` changed the default quorum write timeout from `1500 ms` to `500 ms`. This was intended to fail fast during network partitions, but it was not tested under high disk latency during post-restart compaction.

When several nodes restarted during the rolling deploy, compaction caused disk latency spikes. Followers intermittently failed to acknowledge writes before the shortened timeout. Leaders interpreted the missed acknowledgements as degraded quorum health and stepped down more frequently than expected.

Client libraries retried failed writes immediately, increasing load on already slow replicas. This retry amplification extended the outage until the timeout configuration was rolled back and temporary client rate limits reduced pressure.

## What Went Well

- Alerts detected the issue within minutes.
- The deploy was paused before all shard groups were affected.
- Rollback completed without requiring data restoration.
- Read availability was preserved for most workloads.
- Replication recovered automatically after write pressure normalized.

## What Went Wrong

- The timeout change was bundled with an unrelated binary deploy.
- Load testing did not include restart-time compaction with degraded disk latency.
- Leader election churn alerts were lower priority than write latency alerts.
- Client retry behavior amplified the incident.
- The deploy system did not automatically halt on regional quorum failure signals.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Separate timeout/configuration changes from binary deploys | Storage Platform | P0 | 2026-07-18 |
| Add compaction-under-load tests to pre-release validation | Storage Platform | P0 | 2026-07-24 |
| Restore write timeout default to `1500 ms` and require review for changes | Storage Platform | P0 | Completed |
| Add automatic deploy halt on quorum failure rate above threshold | Release Engineering | P1 | 2026-07-31 |
| Add jittered exponential backoff to default write retries | Client Libraries | P1 | 2026-08-02 |
| Create high-priority alert for leader election churn during deploys | SRE | P1 | 2026-07-25 |
| Add disk latency fault injection to staging cluster tests | SRE | P2 | 2026-08-09 |
| Document rollback procedure for quorum-related incidents | Incident Management | P2 | 2026-07-29 |

## Prevention

Future changes to quorum timing, retry behavior, compaction scheduling, or leader election thresholds will require targeted load testing and explicit storage architecture approval. Deploy automation will also be updated to stop regional rollouts when quorum write failures exceed the defined safety threshold.