# Building a Predictable Distributed Key-Value Store

A distributed key-value store looks simple at the API boundary: clients send `PUT`, `GET`, and `DELETE` requests using opaque keys. The complexity lives underneath that interface. Once data spans multiple machines, the system must decide where each key belongs, how many replicas to maintain, and what to do when nodes fail or networks partition.

## Partitioning and Replication

Consistent hashing assigns keys to partitions while limiting data movement when capacity changes. Each partition has a leader and two followers distributed across failure domains. The leader orders writes through a replicated log, and an operation succeeds only after a quorum confirms it. This provides strong consistency while tolerating the loss of one replica.

```yaml
replication_factor: 3
write_quorum: 2
read_quorum: 2
request_timeout_ms: 750
```

## Reads, Writes, and Failure Recovery

Writes include a monotonically increasing version so replicas can reject stale updates. Reads normally go to the leader, but followers may serve explicitly stale reads for latency-sensitive workloads. When a replica returns after an outage, it replays missing log entries; if the gap is too large, it installs a snapshot and then applies newer entries.

Membership changes deserve the same care as data operations. A coordinator should not immediately replace a node after one missed heartbeat, because transient packet loss can trigger unnecessary rebalancing. Instead, failure detection combines repeated probes with a stabilization window, and partition ownership changes are committed through the consensus layer.

## Operating the Store

Production monitoring should emphasize tail latency, quorum failures, replica lag, disk saturation, and partition imbalance. Compaction and snapshotting must be rate-limited so maintenance does not overwhelm foreground traffic. The most useful operational guarantee is not that failures never happen, but that recovery is bounded, observable, and testable under realistic fault injection.