The Registry Routing Campaign: Two of Nine, and Why It Goes One at a Time

Eighty-seven of ninety-four images in the cluster were pulled straight from upstream registries. Every one of them was still scanned at runtime by Trivy Operator, so there was no visibility gap — what was lost was scan-on-push, registry-side SBOMs, insulation from upstream outages, and, most importantly, Copacetic patchability, because the nightly patch job discovers candidates through the Harbor API alone. Closing that gap is a campaign: nine registries, each needing three separate jobs, each with a blast radius you choose.

Routing a registry through Harbor is a lane change, not a rebuild. Two lanes are done; seven remain.

Three jobs, usually confused

The runbook is emphatic about the middle step, because it is the one people skip. Creating the project is not enough on its own — nothing asks Harbor for those images until the rewrite rule is live. And B alone changes nothing either, because the policy sets background: false: only pods created after the merge are rewritten. Job C is what actually moves traffic.

Why one registry at a time

The first version of the routing config was a single file containing all nine registries. That was wrong in a way that only becomes obvious when you think about failure: enabling it would have activated eighteen rules at once, so eight registries would have been rewritten to Harbor projects that did not exist — and every affected pod on the cluster would have gone into ImagePullBackOff simultaneously. Splitting it into one file per registry, listed commented-out in the kustomization, is what makes an incremental rollout physically possible.

The runbook adds a second discipline: merge each enable on its own. Doing A+B for several registries in one batch triggered a cluster-wide reconcile that exhausted etcd once. One MR per registry, watch for ten minutes, then the next.

The runbook table that separates the jobs — and the note that A alone and A+B alone both change nothing.

Safest first, not biggest first

The per-registry table is sorted safest-first: gcrmirror (2 images, touches only your own scanner) before quay (9 images, touches OpenBao's StatefulSet). The reasoning is explicit: do the smallest one first, make your mistakes where they are cheapest, and only then touch the namespace that holds every secret in the cluster. Two registries are done; the retrospective post at the end of this series tallies what the campaign bought and what it cost.

The estate that motivated it

The routing campaign did not start from a hunch; it started from a measured table. On 2026-09-12 the registry attribution question was settled once and for all — reading the kubelet, not the scan reports — and the numbers were: 87 of 94 distinct running images pulled straight from an upstream registry, 7 through Harbor. Full runtime scanning meant the 87 were never invisible; what was missing was everything a registry can do that a scanner cannot.

The per-registry breakdown turned the problem from "route everything" into a plan:

Two columns of the table are themselves lessons. "Must stay direct" is not an estimate — it is a list of namespaces with a written reason each (the exclusion table in the ceiling post). And "routable" is smaller than "total" for the big three because several images run in both a safe and an excluded namespace, and the safe run already gives them scan-on-push. The real ceiling is 56, not 94, and knowing that before starting is what kept the campaign from promising something architecture would not deliver.

The runbook that makes it repeatable

The per-registry procedure fits on one page and ends in a definition-of-done checklist that audits itself:

• endpoint TEST CONNECTION passed;
• project is Public with Proxy Cache enabled;
• "Automatically scan images on push" ticked (and SBOM-on-push where the etcd quota allows);
• in-cluster pull test shows 0 auth errors;
• clusterpolicy rewrite-registry-<proj> is READY=True;
• a canary workload runs on a harbor.private.example.com/<proj>/... reference;
• Harbor shows a scan result for that repository;
• kubectl get pods -A | grep ImagePullBackOff is empty.

Eight checkboxes, each mapped to a job (A, B or C) and each mapped to a mistake that was actually made: the Private project, the skipped scan-on-push, the laptop-tested pull, the batched MR. That is the pattern the whole campaign runs on — a procedure is not finished when it works; it is finished when the failure modes it survived are written into it.

What "two of nine" leaves

gitlabcom (8 images, ~1,195 fixable Critical+High) and dockerhub phase 1 (12 more images via the existing proxy cache) are done. ghcr and quay are next — the two biggest remaining blocks, and the two that carry the most delicate restarts (OpenBao's StatefulSet among them). The last four registries are one image each and are explicitly scoped as completeness work. The honest summary: roughly two-thirds of the reachable CVE weight is already behind Harbor, and the remaining third is scheduled, staged, and written down.

Next: public project or nothing — the proxy-cache setting that took two workloads down.