# Distributed Key-Value Store — Design Review

**Date:** July 15, 2026  
**Attendees:** Maya Chen, Priya Nair, Daniel Ortiz, Samir Patel, Elena Rossi, Noah Williams  
**Facilitator:** Maya Chen  
**Note-taker:** Noah Williams

## Agenda

1. Review consistency and availability requirements
2. Select replication and partitioning strategies
3. Define failure handling and recovery behavior
4. Agree on observability and rollout plans

## Discussion Notes

- The primary workload is approximately 80% reads and 20% writes, with occasional traffic spikes during batch imports.
- Most consumers can tolerate eventual consistency, but configuration and lease records require strongly consistent conditional writes.
- The team reviewed leader-based replication and leaderless quorum replication. Leaderless replication would improve write availability but add conflict-resolution complexity.
- Consistent hashing with virtual nodes remains the preferred partitioning approach. Virtual-node ownership must account for nodes with different storage capacities.
- Rebalancing should be rate-limited to avoid saturating network links or increasing tail latency.
- Read repair will correct stale replicas during normal traffic. Anti-entropy scans are still required for cold keys.
- Tombstones must be retained long enough to prevent deleted values from reappearing when an offline replica rejoins.
- Operators need per-partition visibility into replica lag, quorum failures, hinted handoff queues, disk usage, and rebalance progress.

## Decisions

- Use consistent hashing with weighted virtual nodes for partition placement.
- Configure a replication factor of three across separate availability zones.
- Use leader-based replication within each partition.
- Default client behavior will provide eventual consistency for reads.
- Support quorum reads and linearizable conditional writes for workloads requiring stronger guarantees.
- Use last-write-wins only for explicitly designated value types; other concurrent updates will return a conflict for application-level resolution.
- Enable hinted handoff with a maximum retention period of 24 hours.
- Run incremental anti-entropy scans continuously, with a full verification pass every seven days.
- Retain tombstones for 14 days initially and revisit the value after collecting outage-duration data.
- Abort automated rebalancing when cluster disk utilization exceeds 80% or p99 latency exceeds the agreed service threshold.
- Roll out to one internal workload before enabling multi-tenant production traffic.

## Action Items

- **Priya:** Draft the consistency API, including quorum-read and conditional-write semantics, by July 22.
- **Daniel:** Prototype weighted virtual-node placement and document distribution variance by July 24.
- **Samir:** Define failure-injection tests covering leader loss, network partitions, stale replicas, and zone outages by July 25.
- **Elena:** Create dashboards and alerts for replica lag, quorum failures, hinted handoff backlog, disk utilization, and rebalance status by July 29.
- **Noah:** Measure the storage and compaction impact of 7-, 14-, and 30-day tombstone retention by July 26.
- **Maya:** Identify the internal pilot workload and confirm its SLOs with the owning team by July 19.
- **Maya and Samir:** Produce rollback criteria and an operational runbook before the pilot launch.

## Open Questions

- Should cross-region replication be asynchronous from launch, or deferred until after the single-region pilot?
- What maximum object size should the API enforce?
- Should clients receive partition-location hints, or should all routing remain server-side?
- What p99 latency threshold should automatically pause rebalancing?