# Distributed Key-Value Store Design Review

**Date:** July 31, 2026  
**Time:** 10:00–11:00 AM ET  
**Facilitator:** Maya Chen  
**Note-taker:** Daniel Ortiz

## Attendees

- Maya Chen — Engineering Manager
- Daniel Ortiz — Staff Engineer
- Priya Shah — Distributed Systems Engineer
- Lucas Martin — Site Reliability Engineer
- Hannah Lee — Storage Engineer
- Omar Haddad — Security Engineer
- Sophie Tremblay — Product Manager

## Agenda

1. Confirm availability and consistency requirements
2. Review partitioning and replication strategy
3. Define failure handling and recovery behavior
4. Discuss operational limits and observability
5. Assign implementation follow-ups

## Discussion Notes

- The initial release will target configuration, feature-flag, and session-metadata workloads. Large-object storage and analytical scans are out of scope.
- Expected traffic is approximately 80,000 reads and 20,000 writes per second, with short bursts up to twice that volume.
- The team agreed that clients need predictable per-key behavior more than cross-key transactions.
- Priya proposed consistent hashing with virtual nodes to distribute partitions and minimize data movement when nodes are added or removed.
- Replication factor three provides sufficient fault tolerance for the first production region. Multi-region replication will be asynchronous.
- Writes will use quorum acknowledgment. Reads will default to quorum but may use a lower-consistency mode for latency-sensitive workloads.
- Conflicting concurrent writes will be resolved using version vectors where practical. A last-write-wins fallback will use server-assigned hybrid logical timestamps.
- Membership and partition ownership will be maintained through a consensus-backed control plane. The data path will remain decentralized.
- Lucas noted that automatic rebalancing must be rate-limited so recovery traffic does not overwhelm healthy nodes.
- Repair will combine read repair with scheduled anti-entropy synchronization using Merkle trees.
- Tombstones will be retained long enough to cover the maximum expected replica outage. The initial retention period is seven days.
- The API will enforce key and value size limits. Proposed defaults are 1 KiB for keys and 1 MiB for values.
- Metrics must include request latency, error rate, quorum failures, replica lag, hinted-handoff backlog, partition skew, disk utilization, and repair progress.

## Decisions

- Use consistent hashing with virtual nodes for partition placement.
- Use a replication factor of three within each region.
- Require two replica acknowledgments for successful writes.
- Default reads to a two-replica quorum; expose an explicitly named eventual-consistency option.
- Provide linearizable compare-and-set operations for a single key; do not support cross-key transactions in the first release.
- Use asynchronous replication between regions.
- Use a consensus-backed service for cluster membership and partition-map updates.
- Support hinted handoff for temporary replica failures.
- Use read repair and periodic Merkle-tree anti-entropy repair.
- Retain tombstones for seven days initially, subject to validation through failure testing.
- Rate-limit partition movement and pause rebalancing automatically when foreground latency exceeds the agreed threshold.
- Encrypt client and inter-node traffic with TLS and encrypt persisted data at rest.
- Launch with 1 KiB key and 1 MiB value limits.

## Open Questions

- Whether the eventual-consistency read mode should be available to all clients or restricted by service policy
- Exact threshold for pausing rebalancing during latency degradation
- Whether version vectors create unacceptable metadata overhead for high-churn keys
- Required recovery-point objective for cross-region failover
- Whether tombstone retention should vary by namespace

## Action Items

- **Priya:** Produce a partitioning and virtual-node simulation using projected node counts and workload skew by August 7.
- **Daniel:** Draft the client API, including quorum options, compare-and-set semantics, retry guidance, and error codes by August 7.
- **Hannah:** Benchmark version-vector metadata overhead and compare it with last-write-wins behavior by August 12.
- **Lucas:** Define service-level indicators, alert thresholds, and rebalancing safety controls by August 12.
- **Omar:** Complete the threat model and recommend certificate rotation and key-management procedures by August 14.
- **Maya:** Schedule a failure-injection review covering node loss, network partitions, disk exhaustion, and regional isolation.
- **Sophie:** Confirm customer requirements for regional failover, consistency controls, and maximum object size by August 10.
- **Team:** Revisit tombstone retention and repair cadence after the first chaos-testing results are available.

## Next Meeting

**Date:** August 14, 2026  
**Topic:** Prototype benchmark results and failure-recovery review