# Distributed Key-Value Store Design Review

**Date:** July 15, 2026  
**Attendees:** Maya Chen, Andre Silva, Priya Nair, Lucas Martin, Elena Petrova, Sam Okafor  
**Facilitator:** Maya Chen  
**Note taker:** Sam Okafor

## Agenda

1. Review consistency and availability requirements
2. Select a replication strategy
3. Define partitioning and rebalancing behavior
4. Discuss failure detection and recovery
5. Confirm observability and rollout plans

## Decisions

- Use consistent hashing with 256 virtual nodes per physical node to distribute keys and limit movement during scaling events.
- Start with a replication factor of three across distinct availability zones.
- Support quorum-based reads and writes, with default settings of `R=2` and `W=2`.
- Offer eventual consistency by default. Strongly consistent conditional writes will use per-key consensus and will be introduced in a later phase.
- Use hinted handoff for temporary replica failures, with hints retained for up to six hours.
- Run anti-entropy repair using Merkle trees. Background repair will be rate-limited to avoid affecting foreground traffic.
- Resolve concurrent writes using hybrid logical clocks and application-visible sibling values. Last-write-wins will not be the default because clock skew could cause silent data loss.
- Detect unhealthy nodes through periodic heartbeats plus phi-accrual failure detection. A node will not be removed from the ring automatically without confirmation from the control plane.
- Persist data using an LSM-tree engine with write-ahead logging. Compaction policies will be tuned separately for write-heavy and read-heavy workloads.
- Rebalancing will stream data incrementally and enforce bandwidth limits. Operators will be able to pause or cancel a rebalance.
- Initial production targets are 99.95% monthly availability, p99 read latency below 20 ms, and p99 write latency below 35 ms within a region.
- Roll out to one internal workload first, followed by a two-week observation period before onboarding additional services.

## Discussion Notes

- Priya noted that quorum settings should remain configurable per namespace because analytics workloads can tolerate stale reads, while session data cannot.
- Andre raised concerns about large sibling sets during network partitions. The group agreed to add a maximum sibling count and emit an alert before applying any fallback resolution.
- Elena recommended separating node liveness from readiness so nodes can recover and warm caches before receiving full traffic.
- Lucas proposed validating repair behavior with injected packet loss and asymmetric partitions, not only complete node failures.
- The team agreed that disk exhaustion is a higher operational risk than CPU saturation. Capacity alerts should account for pending compaction and hinted-handoff data.
- Cross-region replication is out of scope for the first release. The design should avoid assumptions that would prevent adding asynchronous regional replication later.

## Action Items

- **Priya:** Document namespace-level consistency options and recommended quorum configurations by July 22.
- **Andre:** Prototype hybrid logical clock metadata and sibling resolution behavior by July 29.
- **Lucas:** Create failure-injection tests covering packet loss, asymmetric partitions, slow disks, and node restarts by August 5.
- **Elena:** Define dashboards and alerts for latency, quorum failures, repair backlog, hint volume, compaction debt, and disk capacity by July 26.
- **Sam:** Draft the consistent-hashing and rebalance protocol specification by July 24.
- **Maya:** Identify the initial internal workload and confirm its latency, durability, and capacity requirements by July 19.
- **Elena and Lucas:** Run a capacity test comparing leveled and size-tiered compaction before the next design review.
- **Maya:** Schedule the follow-up review for the week of August 10.

## Open Questions

- What maximum object size should be supported in the first release?
- Should tombstone retention be fixed globally or configurable per namespace?
- What sibling-count limit provides acceptable metadata overhead without causing premature conflict resolution?
- How should repair traffic be prioritized during an active availability-zone failure?
- Which authentication and authorization model will be required before external service adoption?