From: Maya Chen <maya.chen@northstarinfra.example>
To: Storage Platform Team <storage-platform@northstarinfra.example>
Subject: Rebalancing plan for the Aurora KV cluster
Date: Tue, 14 Jul 2026 09:42:17 -0400

Team,

After reviewing last night's load test, I think we should proceed with the
incremental rebalance rather than taking the Aurora key-value cluster through a
full ring rebuild.

The hot partitions are mostly concentrated in the customer-session namespace,
and the p99 read latency only spikes when replica 3 in us-east-1b takes
ownership of more than 14% of the token range. Moving 3-4% of that range to the
new r7g nodes should be enough to get us back under the 18 ms target without
creating a large compaction backlog.

Proposed sequence:

1. Add the six new nodes as non-voting replicas.
2. Stream token ranges in 512 MB chunks during the low-traffic window.
3. Promote the new replicas once hinted handoff drains below 10k entries.
4. Re-run the mixed read/write workload before removing any old capacity.

Please do not disable read repair during this. The last incident showed that
the extra repair traffic is cheaper than debugging stale cart state after a
partial handoff.

> Can we just increase the cache on the current nodes and avoid moving data?
>
> - Eli

We can raise cache by another 8 GB, but it only helps the product-metadata
namespace. The session keys churn too quickly, so the hit rate barely improves.
The safer fix is to spread the write load.

If nobody objects, I'll schedule the first range move for Wednesday at 02:00
UTC and post the runbook before end of day.

Maya