# Designing a Distributed Key-Value Store

A distributed key-value store looks simple at the API boundary: `put(key, value)`, `get(key)`, and `delete(key)`. The hard part is making those operations predictable when disks fail, nodes restart, networks partition, and traffic shifts unevenly across the cluster. A production design usually starts by defining the failure model, latency target, and consistency guarantees before choosing storage engines or replication algorithms.

## Partitioning and Replication

Most systems split the keyspace into shards using consistent hashing or range partitioning. Each shard is assigned to multiple replicas, commonly three, spread across failure domains such as hosts, racks, or zones. Consistent hashing makes node additions less disruptive, while range partitioning can make scans easier but requires careful split and merge logic to avoid hot partitions.

## Consistency Model

A common approach is leader-based replication per shard: writes go to the shard leader, then replicate to followers through an ordered log. Reads can go to the leader for stronger consistency or to followers for lower latency with possible staleness. Some stores expose quorum settings so clients can tune availability and consistency per workload.

```yaml
replication_factor: 3
write_quorum: 2
read_quorum: 2
compaction:
  strategy: leveled
  target_file_size_mb: 128
```

## Storage and Compaction

On each node, an LSM-tree is a practical storage layout for write-heavy workloads. Writes land in a memory table and append-only commit log, then flush into immutable sorted files. Background compaction keeps read amplification under control, but it must be rate-limited because aggressive compaction can steal IO from foreground reads and writes.

## Operational Concerns

The store needs more than replication to be reliable. Operators need metrics for tail latency, quorum failures, leader elections, disk saturation, compaction backlog, and replica lag. Repair jobs should continuously compare replicas and fix divergent data, while rebalancing should move shards gradually so recovery does not create a second outage.

## Final Shape

A realistic distributed key-value store is less about a clever hash map and more about bounding uncertainty. The API should be small, but the internals need explicit choices around placement, replication, consistency, storage layout, and observability. When those tradeoffs are documented and exposed carefully, the system becomes easier to reason about during both normal growth and failure recovery.