# Meeting Notes: Distributed Key-Value Store Design Review

Date: 2026-07-10  
Time: 10:00 AM - 10:50 AM ET  
Location: Engineering Zoom

## Attendees

- Maya Chen, Staff Backend Engineer
- Luis Romero, Distributed Systems Engineer
- Priya Nair, SRE Lead
- Evan Brooks, Storage Engineer
- Hannah Kim, Product Manager
- Omar Haddad, Security Engineer
- Sofia Martinez, QA Engineer

## Agenda

1. Review current prototype performance and failure behavior
2. Confirm replication and consistency model for v1
3. Discuss partitioning, rebalancing, and node membership
4. Identify operational requirements for launch
5. Assign follow-up work

## Discussion Notes

- The prototype sustained 42k writes/sec and 78k reads/sec in the 6-node test cluster using 1 KB values and RF=3.
- Tail latency remains higher than target during compaction; p99 write latency peaked at 185 ms under sustained load.
- Team agreed that v1 should prioritize predictable operations and data safety over maximum throughput.
- Read repair will be asynchronous for now to avoid adding latency to foreground reads.
- Priya raised concerns about noisy rebalancing during node replacement. Luis proposed limiting token movement per node and exposing operator controls.
- Omar noted that inter-node traffic needs mutual TLS before any shared staging environment rollout.
- Sofia asked for deterministic failure tests covering coordinator crash, replica timeout, and hinted handoff replay.

## Decisions

- Use consistent hashing with virtual nodes for partition assignment.
- Launch v1 with replication factor 3 as the default.
- Support quorum reads and quorum writes as the default consistency level.
- Keep last-write-wins conflict resolution for v1, using server-side hybrid logical timestamps.
- Implement hinted handoff, but cap hint retention at 24 hours.
- Make automatic rebalancing opt-in for the first production release.
- Require mutual TLS for all node-to-node communication.
- Defer multi-region active-active replication until after single-region production validation.

## Action Items

- Luis: Draft the partition ownership and rebalancing design doc by 2026-07-15.
- Evan: Investigate compaction-related p99 latency spikes and propose tuning changes by 2026-07-17.
- Priya: Define launch SLOs, alert thresholds, and runbook outline by 2026-07-16.
- Omar: Provide mTLS certificate rotation requirements by 2026-07-14.
- Sofia: Add failure-mode test cases for coordinator crash, replica timeout, and hinted handoff replay by 2026-07-18.
- Maya: Review quorum behavior and timestamp conflict handling with the storage team by 2026-07-15.
- Hannah: Confirm customer-facing durability and availability language for the v1 beta by 2026-07-19.

## Open Questions

- Should operators be able to temporarily pin partitions during maintenance?
- What is the acceptable maximum recovery time after a single-node failure?
- Do we need per-tenant request throttling in v1, or can it wait until beta feedback?