From: Maya Chen <maya.chen@northbridge.io>
To: Storage Platform Team <storage-platform@northbridge.io>
Subject: Follow-up on KV store read repair and quorum behavior
Date: Fri, 10 Jul 2026 14:37:22 -0400

Hi team,

I finished reviewing the staging results for the distributed key-value store after yesterday's changes to hinted handoff and read repair.

The good news is that p99 read latency stayed under 18 ms during the three-node failure simulation, even with the extra digest reads enabled. Write availability also looked healthy: the client library correctly accepted writes at W=2 and queued hints for the unavailable replicas without backing up the coordinator.

There are still two issues we should address before promoting this build:

1. Tombstone-heavy partitions caused several read repairs to exceed the 250 ms timeout.
2. During node recovery, hinted handoff replay briefly saturated one shard and delayed compaction by about six minutes.

I think we should cap hint replay per destination shard and make read repair skip stale tombstones that are already past gc_grace_seconds. That should reduce the recovery spike without weakening consistency guarantees.

For reference, this was Sam's note from the earlier thread:

> If we keep R=2/W=2 in production, we can tolerate one replica being down, but read repair needs to stay opportunistic. It should not turn a normal user read into a full anti-entropy pass.

That matches what I saw in the traces. The consistency model is behaving as expected, but the repair path is doing too much work on cold partitions.

I'll open a patch with the replay throttle this afternoon.

Maya