# Operation Longplay 2026-08-24: mishaps log

Second campaign. The harness itself behaved; most of what follows is
infrastructure and measurement, and one item that invalidates a metric.

## Silently produced wrong data

**The encode phase timer measures parking, not CPU.** `encode_blob`
(`sdk/src/track/write.rs:407`) starts a timer, awaits
`spawn_blocking(prepare_plan)`, then stops it. `spawn_blocking` runs regardless
of polling, so the CPU work finishes on time while the timer runs until the
handle is awaited. The stream stage only polls those futures at
`in_flight.next().await` and parks in between on a depth-2 channel
(`ENCODE_AHEAD = 2`), so a slow `store` inflates `encode`. At 1 GiB, Singapore
recorded 32.90 ms/MiB against US at 10.49 on identical c4d-standard-8 Turin
hardware. Nothing errored and the number looks like a CPU result. It is not one.
Symptom to watch for: encode inflation tracking store inflation (Singapore
2.79x vs 3.31x), and the effect appearing only once chunk count exceeds worker
count.

**Two machine types were compared as if they were regions.** Jakarta has no c4d
family at all, so the first Asia leg ran t2d (Milan) against c4d (Turin)
everywhere else, and the 3x encode gap read as an Asia result. It was silicon.
The Singapore re-run on matching Turin is what separated them. Never let the
machine type vary with the variable under test without saying so up front.

## Tooling that lies

**rtk mangles ripgrep output.** `rg -n "spawn_blocking|block_in_place"` printed
`tokio::task::n(move || prepare_plan(owned))`, replacing the matched text with
`n`. Reading that literally would mean the code does not use `spawn_blocking`,
which is the opposite of the truth and would have killed the correct diagnosis.
Same class as the 2026-07-29 `rg -r` entry. Route source greps through
`rtk proxy`.

**A completion monitor raced its own log.** The Jakarta watcher grepped the log,
found no `done.`, then checked the pid, which had just exited, and reported
`PROCESS GONE without done marker`. The run had completed normally, 78 of 78
rows. Re-read the log after observing the process exit, before declaring failure.

## Infrastructure

**GCP CPUS_ALL_REGIONS is 12 for the whole project and is not self-service.** A
preference asking 33 was filed 2026-07-25 and Google granted 12 on 2026-07-29;
the quota reports `ineligibilityReason: NOT_ENOUGH_USAGE_HISTORY`. Three
8 vCPU clients need 24, so the legs ran one at a time, each box deleted before
the next region was created. Region quota is irrelevant: Singapore shows 32
CPUs free and still cannot be used past the global cap.

**c4d is not everywhere.** Stocked out in us-central1-a, -b, -c and -f before
us-east1-c took it. Absent entirely from asia-southeast2 (Jakarta), where C4
quota is also 0. In Asia-Pacific, c4d-standard-8 exists only in
asia-southeast1, asia-east1, asia-northeast1, asia-south1 and
australia-southeast2.

**c4d needs a hyperdisk boot disk.** pd-balanced is rejected. t2d takes
pd-balanced normally.

**scp before sshd is up fails with connection refused**, and gcloud does not
wait. Gate provisioning on `until gcloud compute ssh … --command=true`.

**Stop, do not delete, while a replacement is unproven.** Quota counts running
vCPU, so stopping frees it just as well and keeps the rollback. This mattered:
the US box took five zone attempts. The 2026-07-29 log records the opposite
mistake costing a whole region.

## Judgement errors

**Acted on a stale memory that said a live box was retired.** The note claimed
the GCP node box was Milan and that Qatar was decommissioned. In fact Milan is
gone and `tape-node-qtr-01` is live, running `tape-node`, `tape-gateway` and
`tape-collector`, and it is the origin behind `gw.miester.id`. It was proposed
as a candidate for deletion to free 2 vCPU. Verify a box is idle by looking at
it, never by trusting a note. Memory corrected.

## Still open

- **R4 does not poll for visibility.** R2 polls until an object is servable and
  records the wait; R4 does one warming fetch and one measured fetch. Two US
  cells 404'd at the edge because the CDN-side gateway's tail lagged the
  origin-side gateway's. R4 needs R2's treatment.
- **The encode pipeline caps at 4 workers** (`MAX_ENCODE_WORKERS`) with a
  depth-2 handoff, so half the cores on an 8 vCPU client are idle on that path.
  Separate from the timer bug, and a real throughput question.
- **No Asia storage node.** Both Asia legs measure the Pacific, not the
  protocol. With Clay n=20 k=7 a read needs 7 of 20 slices, so a single Asia
  node would supply at most one of them. A meaningful Asia region row needs
  roughly a third of the fleet in Asia, not one box.
- **Jakarta's read transit is anomalous.** Near-identical store cost to
  Singapore, six times worse cold gateway read. Worth a traceroute before it is
  attributed to geography.