# Building a Distributed Key-Value Store That Survives Failure

A distributed key-value store looks simple at the API boundary: clients send `PUT`, `GET`, and `DELETE` requests against opaque keys. The difficulty begins once data must remain available across machines, zones, and routine failures. A practical design usually partitions the keyspace with consistent hashing, assigns each partition to several replicas, and routes requests through any node capable of locating the current partition owner.

## Replication and Consistency

Replication determines both durability and the consistency model visible to clients. With a replication factor of three, a write coordinator might wait for acknowledgements from two replicas before reporting success. Reads can follow the same quorum pattern, allowing the system to tolerate one unavailable node while reducing the chance of returning stale data. This trade-off is commonly exposed through tunable request settings:

```yaml
replication_factor: 3
write_quorum: 2
read_quorum: 2
request_timeout_ms: 750
```

Quorums do not eliminate conflicting writes. Network partitions can allow separate coordinators to accept updates to the same key, especially when availability is prioritized. Each stored value therefore needs version metadata, such as a hybrid logical timestamp or version vector. The system can resolve conflicts with last-write-wins semantics, merge application-defined values, or return sibling versions for the client to reconcile.

## Failure Detection and Repair

Nodes should treat failure detection as an estimate rather than proof. A gossip protocol can distribute membership and health information without relying on a central controller, but transient packet loss must not immediately trigger expensive rebalancing. Suspicion timers and failure thresholds help distinguish a dead node from a slow one. When a replica is temporarily unreachable, hinted handoff can retain its writes elsewhere and replay them after recovery.

Background repair is essential because successful quorum operations can still leave individual replicas inconsistent. Read repair fixes divergent values discovered during normal traffic, while periodic anti-entropy jobs compare partition summaries—often Merkle trees—and synchronize only mismatched ranges. Tombstones for deleted keys must remain long enough to reach every replica; removing them too early can resurrect stale values during repair.

## Operating the System

Production reliability depends as much on observability and capacity planning as on replication algorithms. Operators should monitor tail latency, quorum failures, compaction backlog, repair progress, disk saturation, and partition skew. Rebalancing should be rate-limited so it does not compete with foreground traffic, and overload controls should reject excess work before queues consume all available memory. Finally, recovery procedures must be tested under realistic conditions: losing a node, isolating a zone, restoring from backup, and replacing corrupted replicas are operational paths, not theoretical edge cases.