# Distributed Key-Value Store Design Review

**Date:** July 15, 2026  
**Time:** 10:00–11:00 AM ET  
**Location:** Zoom  
**Facilitator:** Priya Shah  
**Note taker:** Daniel Kim  

## Attendees

- Priya Shah — Engineering Manager
- Daniel Kim — Staff Backend Engineer
- Marta López — Distributed Systems Engineer
- Jonah Reed — Site Reliability Engineer
- Aisha Bello — Security Engineer
- Eric Wu — Product Manager
- Lena Fischer — Data Platform Engineer

## Agenda

1. Review consistency and availability requirements
2. Confirm partitioning and replication strategy
3. Discuss failure detection, recovery, and rebalancing
4. Define observability and operational safeguards
5. Agree on MVP scope and rollout plan

## Discussion Notes

- The initial workload is expected to be 80% reads and 20% writes, with values typically under 64 KB.
- The service must remain available during a single-node or single-availability-zone failure.
- Strong consistency is required for conditional writes and lease records. Eventual consistency is acceptable for ordinary reads when explicitly requested by clients.
- The team discussed leaderless replication but decided its conflict-resolution and repair complexity would add risk to the first release.
- Keys will be distributed using consistent hashing with virtual nodes. Capacity-weighted token allocation will support heterogeneous instances.
- Each shard will use three replicas across separate availability zones. One replica will act as leader and serialize writes.
- Default reads and writes will use quorum semantics. Clients may request a stale read from a local follower for lower latency.
- Membership changes will be coordinated through a strongly consistent control plane. Nodes will not independently add or remove cluster members.
- Failed replicas will recover from snapshots plus the replicated log. Merkle-tree-based anti-entropy repair is deferred until operational data shows it is necessary.
- Rebalancing must be rate-limited and pausable to avoid saturating disks or network links.
- Hot keys remain a concern. The first release will expose per-key sampling and shard-level load metrics; automatic hot-key splitting is out of scope.
- Data at rest will be encrypted using the platform key-management service. Service-to-service traffic will use mutual TLS.
- The target service-level objective is 99.95% monthly availability, with p99 latency targets of 20 ms for reads and 50 ms for writes within one region.

## Decisions

- Use a leader-based replication model with three replicas per shard across three availability zones.
- Use consistent hashing with virtual nodes and capacity-weighted placement.
- Require quorum acknowledgement for writes and linearizable reads.
- Offer an opt-in stale-read mode served by followers.
- Keep cluster membership and placement decisions in a strongly consistent control plane.
- Use snapshot-and-log recovery for the MVP.
- Rate-limit and expose pause controls for replica movement and shard rebalancing.
- Limit the MVP to a single region; cross-region replication will be designed separately.
- Enforce a 64 KB value limit at launch, with larger-object support deferred.
- Launch behind an allowlist and expand only after failure-injection and recovery testing passes.

## Action Items

- **Daniel:** Draft the shard replication and leader-election design document by July 22.
- **Marta:** Prototype consistent-hash placement and measure key movement during node additions and removals by July 24.
- **Jonah:** Define dashboards, alerts, and runbooks for replica lag, quorum loss, disk pressure, and rebalance progress by July 29.
- **Aisha:** Complete the threat model and review encryption, certificate rotation, and authorization requirements by July 26.
- **Lena:** Build a workload generator covering read-heavy traffic, hot keys, and 64 KB values by July 25.
- **Eric:** Confirm client requirements for stale reads, conditional writes, and expected dataset growth by July 19.
- **Priya:** Schedule the failure-injection review and identify two pilot teams by July 30.
- **Daniel and Marta:** Propose API error semantics for timeouts, unavailable quorums, and version conflicts by July 23.

## Open Questions

- Should conditional writes use explicit version numbers, opaque ETags, or both?
- How long should tombstones be retained before compaction?
- What limits should apply to per-tenant request rates and storage usage?
- Is follower-read staleness bounded by time, log position, or both?
- When should anti-entropy repair become mandatory rather than optional?

## Next Meeting

**Date:** July 30, 2026  
**Topic:** Prototype results, failure testing plan, and API review