# What is actually slow: node-side deep dive, 2026-08-24

Scanned `tape-node-qtr-01` (GCP me-central1, e2-medium) and `tape-hz-02`
(Hetzner, 4 node instances) during and after live 1 GiB writes.

## The fleet nodes are not the bottleneck

`tape-hz-02` during active 1 GiB writes from a us-east1 client:

| metric | value |
|--------|-------|
| disks | 4x Toshiba MG09 14.6 TB, rotational |
| peak %util | 1.46% |
| iowait | 0.04 - 0.19% |
| w_await | 2 - 18 ms |
| load average | 0.35 |
| throttle/reject log lines, 30 min, 4 instances | 2, 1, 2, 0 |
| WARN+ERROR, 3 h, per instance | 17 - 23 |

Spinning disks, and they are idle. Nothing in the logs shows the serving side
refusing or delaying work. The nodes could take far more than we are sending.

## Large writes are egress-bandwidth-bound

Clay `n=20, k=7` puts **2.86 bytes on the wire per byte written**, exactly and
invariantly. Dividing coded bytes shipped by the write wall:

| region | 50 MiB | 512 MiB | 1 GiB |
|--------|--------|---------|-------|
| us | 0.19 Gbps | 0.76 | **0.96** |
| eu | 0.17 | 0.59 | **0.83** |
| sg | 0.12 | 0.34 | **0.37** |
| jakarta | 0.11 | 0.29 | **0.35** |

Throughput climbing with size is the signature of a fixed cost at the bottom and
a ceiling at the top. US tops out at 0.96 Gbps, which is close enough to a 1 Gbps
line rate to be worth confirming as a real cap.

A 1 GiB write ships 3.07 GB. At 120 MB/s that is 25.6 s, and the measured wall
is 25.58 s. The write is doing exactly as well as the link allows.

## There is a crossover, and it is near 512 MiB

Two floors compete, and the larger one wins:

| size | chain floor | bandwidth floor (us) | binds |
|------|-------------|----------------------|-------|
| 50 MiB | ~6.0 s | 1.25 s | chain |
| 512 MiB | ~10.5 s | 12.8 s | roughly even |
| 1 GiB | ~17.2 s | 25.6 s | bandwidth |

So "what is slow" has two different answers. Below ~512 MiB it is chain
confirmations: register, certify_submit and certify_visible, which cost about
6 s no matter the payload or the continent. Above it, it is bytes over the wire
times 2.86.

This also explains why the pipeline-depth sweep barely moved the median: you
cannot pipeline past a saturated link. The buffers were never the constraint at
1 GiB, and the chain was never the constraint either.

## The Qatar box is misconfigured

It is the only box pointed at public RPC instead of the fleet cache.

| box | rpc |
|-----|-----|
| tape-hz-02 | `http://167.172.28.146:8899?api=…` (fleet cache) |
| tape-node-qtr-01 | Helius primary, rpcpool secondary |

Consequences measured over one hour:

- `tape-node`: 560 `getBlock http 429: rate limited`
- `tape-gateway`: 664 rate-limit warnings, 679 total RPC warnings
- `tape-collector`: 0

Three services on one 2 vCPU box, each replaying blocks against the same public
endpoints. Both the node and the gateway still report `lag=0` and dispatch
45-49 blocks per 10 s, so the retries are absorbing it and this is **not** a
sync failure. What it costs is wasted RPC budget, added latency on every block
fetch, and a noisier tail.

That tail matters: this box is the origin behind `gw.miester.id`, and the
campaign's only two failed cells were R4 404s where this gateway's object tail
lagged the DigitalOcean gateway's. Rate-limited block fetching is a plausible
contributor.

The cache answers from this box in 411 ms (`getSlot`, http 200), so the fix is a
config edit plus a restart, not a network problem. Runbook invariant already
covers it: point the fleet at the cache.

## Ranked

1. **Repoint the Qatar box at the fleet cache.** Config-only, removes ~1200
   rate-limit events an hour across two services.
2. **Confirm the client egress ceiling.** If us-east1 really is on a 1 Gbps cap,
   every large-write number in the campaign is a measurement of that cap and not
   of Tapedrive. This changes how the results should be presented.
3. **Nothing on the storage nodes.** They are idle on spinning disks. Do not
   spend effort there until something else moves.

## Addendum: client-side ceilings ruled out by measurement

Two hypotheses from the first pass were tested and both failed.

**"US is pinned at ~1 Gbps egress."** False. The client box measures 3.23 Gbps
uplink across 8 streams to Cloudflare, against 1.02 Gbps used by the best 1 GiB
write. A single upload also leaves fleet boxes at 20-25% of their 1 Gbps NIC.
No link is saturated by one uploader.

**"Client TCP tuning will recover the per-stream gap."** False. 30 writes at
1 GiB, `buf8` pipeline throughout, arms interleaved:

| arm | cc | p50 | p90 | min | vs stock | t |
|-----|----|-----|-----|-----|----------|---|
| stock (`slow_start_after_idle=1`, wmem 4 MB) | cubic | 25.76 s | 28.74 s | 22.86 s | — | — |
| tuned (`=0`, wmem 16 MB) | cubic | 24.12 s | 28.69 s | 20.04 s | -474 ms | -0.32 |
| bbr (`=0`, wmem 16 MB) | bbr | 24.66 s | 27.16 s | 19.97 s | -913 ms | -1.04 |

Neither reaches significance. The tuning doc's network settings are sound for
the nodes, but on the client they do not move a 1 GiB write.

## What is left, and the evidence for it

Per-slice upload runs ~2.3 MB/s while a single stream to the same Hetzner
network measures 23 MB/s. The client is not gated: `slice_concurrency` is
`GROUP_SIZE` (20) with one semaphore permit per member, so a chunk's 20 slices
are genuinely parallel, ~48 streams in flight with chunks overlapped.

The remaining difference is that a slice upload is not a transfer. The client's
timer covers the node's whole `put_slice`: deserialize copy, four store reads,
SHA-256 over the slice, a synchronous disk write and a BLS signature, none of
which is offloaded from the async runtime.

Supporting signal: R1 reads pull slices at ~7.1 MB/s per stream against
~2.3 MB/s for writes, a 3x gap on the same paths and the same peers. Reads do
far less node-side work per byte. That is consistent with node processing being
a real share of upload latency rather than a latent concern.

This promotes the blocking-handler item from "fix before scaling" to "the next
thing worth measuring". Confirming it needs node-side timing around `put_slice`,
which does not exist today.