# Building a Distributed Key-Value Store That Survives Failure

A distributed key-value store looks simple at the API boundary: clients send `PUT`, `GET`, and `DELETE` requests using opaque keys. The complexity begins once data must remain available across machines, regions, and routine failures. A practical design partitions the keyspace with consistent hashing, assigns each partition to several replicas, and routes requests through any node that can locate the current partition owner.

## Replication and Consistency

Each partition uses a small Raft group with three or five replicas. The leader serializes writes into a replicated log and acknowledges a request only after a quorum has persisted the entry. Reads can go through the leader for linearizable semantics, while latency-sensitive workloads may use follower reads with an explicit staleness bound. Exposing this tradeoff in the client API is safer than silently returning potentially outdated values.

```yaml
replication_factor: 3
write_quorum: 2
read_consistency: linearizable
request_timeout_ms: 750
```

## Storage and Compaction

On each node, writes first enter a write-ahead log and an in-memory sorted table. When the table reaches its size threshold, it is flushed as an immutable SSTable. Background compaction merges SSTables, removes overwritten values, and eventually discards tombstones. Compaction must be rate-limited because unrestricted merging can saturate disks and cause request latency to spike even when CPU and memory appear healthy.

## Rebalancing Without an Outage

Membership changes are coordinated through versioned cluster metadata. When a node joins, partitions are copied to it before routing changes take effect; when a node leaves, replicas are replaced before ownership is removed. Transfers use snapshots followed by incremental log replay so that partitions remain writable during migration. Limiting concurrent transfers prevents a routine scale-out operation from overwhelming the network.

## Operating the System

Useful monitoring focuses on quorum health, replication lag, leader changes, compaction backlog, disk latency, and request percentiles by operation. Operators also need automated repair for missing replicas and periodic checksum comparisons to detect silent corruption. Failure testing should include unavailable zones, full disks, delayed packets, expired certificates, and nodes restarting with stale metadata—the mundane conditions that usually expose more defects than clean process crashes.