# Building a Practical Distributed Key-Value Store

A distributed key-value store looks simple at the API boundary: clients send `PUT`, `GET`, and `DELETE` requests using opaque keys. Internally, however, every request must be routed to the correct nodes, replicated across machines, and handled safely during failures. A practical design starts by partitioning the keyspace with consistent hashing, which limits data movement when nodes join or leave the cluster.

## Replication and Consistency

Each partition is replicated to three nodes, with one replica acting as leader. Writes go through the leader, which records the mutation in a write-ahead log before forwarding it to followers. A write is acknowledged after a quorum confirms persistence, providing durability without waiting for every replica. Reads can use the leader for strong consistency or a nearby follower when lower latency matters more than immediately observing the latest write.

```yaml
replication_factor: 3
write_quorum: 2
read_consistency: leader
request_timeout_ms: 750
```

## Failure Handling

Nodes exchange periodic heartbeats, but missed heartbeats alone do not trigger reassignment because transient network congestion can resemble a failure. Instead, the cluster uses a consensus-backed membership service to confirm topology changes and assign a new leader. During a network partition, only the side holding a quorum may accept writes, preventing replicas from developing conflicting histories.

## Storage and Compaction

On each node, incoming mutations are appended to a log and indexed in memory. Immutable sorted files are flushed to disk once the memory table reaches its configured limit. Background compaction merges these files, discards overwritten values, and eventually removes tombstones created by deletes. Rate-limiting compaction is important because an aggressive merge can saturate disk bandwidth and cause sharp increases in request latency.

## Operating the Cluster

Production reliability depends as much on observability as on replication. Operators should track quorum failures, replication lag, compaction backlog, disk utilization, and latency percentiles by operation type. Capacity planning must also reserve room for rebuilding replicas after a node loss; a cluster running near full disk or network capacity may remain online during a failure but recover too slowly to tolerate the next one.