2026-07-11 09:14:03 EDT | Customer (Maya Chen): Hi, we're seeing intermittent stale reads from our distributed key-value store after a node replacement last night.
2026-07-11 09:14:21 EDT | Support (Arun Patel): Thanks for reaching out. Can you share the cluster size, replication factor, and consistency levels for reads and writes?
2026-07-11 09:14:48 EDT | Customer (Maya Chen): 9 nodes across 3 AZs, RF=3. Writes are QUORUM, reads are LOCAL_QUORUM.
2026-07-11 09:15:10 EDT | Support (Arun Patel): Understood. Are the stale reads isolated to keys whose replicas include the replaced node?
2026-07-11 09:15:37 EDT | Customer (Maya Chen): That seems likely. Most reports involve token ranges owned by node kv-07, which we replaced.
2026-07-11 09:16:05 EDT | Support (Arun Patel): Please check whether bootstrap completed and whether repair ran after the replacement. What does `nodetool status` show for kv-07?
2026-07-11 09:16:42 EDT | Customer (Maya Chen): kv-07 is UN and owns 11.4%. Bootstrap says completed at 02:18. We did not run a full repair afterward.
2026-07-11 09:17:18 EDT | Support (Arun Patel): That may explain it. If kv-07 streamed older SSTables or missed hinted handoff windows, some replicas can disagree until read repair or anti-entropy repair catches up.
2026-07-11 09:17:50 EDT | Customer (Maya Chen): Shouldn't QUORUM writes prevent this?
2026-07-11 09:18:22 EDT | Support (Arun Patel): QUORUM writes protect acknowledged writes going forward, but a replaced node can still have gaps from historical data unless token ranges are fully synchronized.
2026-07-11 09:18:57 EDT | Customer (Maya Chen): We're also seeing elevated read latency, p99 around 840 ms.
2026-07-11 09:19:28 EDT | Support (Arun Patel): That can happen if read repair is triggering frequently. Can you confirm the read repair rate and pending compactions?
2026-07-11 09:20:11 EDT | Customer (Maya Chen): Read repair attempts jumped from 200/min to 4,800/min. Pending compactions on kv-07 are at 312.
2026-07-11 09:20:49 EDT | Support (Arun Patel): That lines up. The node is likely serving while catching up. I recommend running an incremental repair for kv-07's token ranges first, then monitoring pending compactions.
2026-07-11 09:21:25 EDT | Customer (Maya Chen): Is it safe to run during peak traffic?
2026-07-11 09:21:54 EDT | Support (Arun Patel): Use throttling. Set stream throughput to 80-120 MB/s and run one keyspace at a time. Avoid running repair cluster-wide right now.
2026-07-11 09:22:37 EDT | Customer (Maya Chen): We have keyspaces `sessions`, `profiles`, and `inventory`. The stale reads are mostly `inventory`.
2026-07-11 09:23:04 EDT | Support (Arun Patel): Start with `inventory`. Also verify that client requests are not using token-aware routing to prefer kv-07 too heavily.
2026-07-11 09:23:44 EDT | Customer (Maya Chen): Driver logs show kv-07 has a higher request count than peers. Maybe the token map refreshed after replacement.
2026-07-11 09:24:17 EDT | Support (Arun Patel): Please restart one application instance or force a metadata refresh in the driver pool. We want clients spreading reads evenly across replicas.
2026-07-11 09:25:02 EDT | Customer (Maya Chen): Done for two app instances. Request distribution looks more balanced now.
2026-07-11 09:25:36 EDT | Support (Arun Patel): Good. Next, run the throttled repair for `inventory` on kv-07 and watch read latency, dropped mutations, and compaction backlog.
2026-07-11 09:26:19 EDT | Customer (Maya Chen): Repair started. Streaming around 92 MB/s. No dropped mutations so far.
2026-07-11 09:31:47 EDT | Customer (Maya Chen): p99 read latency is down to 410 ms. Stale read reports have slowed but not stopped.
2026-07-11 09:32:20 EDT | Support (Arun Patel): Expected while repair is still reconciling ranges. Are you seeing digest mismatches decrease?
2026-07-11 09:32:58 EDT | Customer (Maya Chen): Yes, digest mismatches dropped from 1,900/min to 360/min.
2026-07-11 09:33:29 EDT | Support (Arun Patel): That's a strong sign the replicas are converging. Keep the repair running and let compaction drain afterward.
2026-07-11 09:34:06 EDT | Customer (Maya Chen): Anything else we should change to prevent this next time?
2026-07-11 09:34:41 EDT | Support (Arun Patel): After node replacement, run targeted repair before returning the node to full production load. Also confirm hinted handoff windows cover the replacement duration.
2026-07-11 09:35:19 EDT | Customer (Maya Chen): Our hinted handoff window is 3 hours. The node was down for almost 5.
2026-07-11 09:35:48 EDT | Support (Arun Patel): Then hints expired for part of that outage. Repair is the correct recovery path here.
2026-07-11 09:42:33 EDT | Customer (Maya Chen): Repair for `inventory` completed. Digest mismatches are near baseline and p99 is 190 ms.
2026-07-11 09:43:01 EDT | Support (Arun Patel): Great. Please run the same targeted repair for `sessions` and `profiles` during a lower-traffic window, then keep kv-07 under observation for 24 hours.
2026-07-11 09:43:31 EDT | Customer (Maya Chen): Will do. Thanks for the help.
2026-07-11 09:43:49 EDT | Support (Arun Patel): You're welcome. I'll leave the case open until tomorrow so you can update us if stale reads return.