Patching Without a List: A Nightly Pipeline That Finds Its Own Targets
Every image list you maintain by hand is stale the day after you write it. Our Copacetic pipeline discovers candidates from Harbor's own scan results instead.
The obvious way to build automated image patching is a list: enumerate the images you care about, run the patcher over them nightly, maintain the list when new images appear.
We built it without a list, and the reason is simple: every maintained list is stale the day after you write it. A newly pushed image, a newly cached proxy image, a team's first container — none of them are on the list, and nobody remembers the list when they ship.
The design
The nightly job runs at 03:15 UTC as a Kubernetes CronJob. Six stages:
Discovery replaces the maintained list — being scanned is what makes you a candidate.
Discovery queries the registry's own scan results. Harbor has already scanned everything pushed to it (with the same Trivy engine the cluster uses elsewhere, so findings are directly comparable). The job asks Harbor's /api/v2.0 for artifacts carrying fixable Critical/High OS CVEs and works on those. Coverage of new images is automatic, because being scanned is what makes you a candidate.
A cap keeps failures bounded. MAX_PER_RUN=10 is a safety valve: a database refresh or a bad push cannot turn the nightly job into a two-hour fleet-wide rebuild, and each run's blast radius is known.
Copy-then-patch. copa has no destination flag — it always pushes back to the source repository, which fails outright on pull-only proxy-cache projects. So crane copies each candidate into a writable patched project first, and the patch happens in place there. Results land as patched/<project>/<repo>:<tag>-patched; source images are never modified.
Exclude via Git, not kubectl. The deny-list (exclude.txt) is a set of POSIX regexes in a ConfigMap managed by GitOps. Editing it with kubectl edit does not work in a Flux cluster — drift detection reverts it within the hour. If you want an image permanently excluded, that is a merge request.
The stage that makes it safe: your tests
The job never promotes. A scheduled job cannot run your application's test suite, and a functional test is the only thing that makes automated patching safe — which is why the fixture smoke test (baseline before patch, same test after) is the single most important test in the entire workflow.
The promotion path is deliberate:
- Patched image lands as
-patchedtag. - You deploy it to staging.
- You run your suite.
- Only then retag as the release.
The original image is untouched and remains the rollback. An automated patch without a human-owned test gate is not automation; it is a nightly gamble with a schedule attached.
What it costs
No Helm chart — a rootless BuildKit deployment, two ConfigMaps, a CronJob with a ServiceAccount. Three binaries are fetched by an init container (copa, crane, jq) because no published image carries both copa and trivy.
The rootless part took the most care: RootlessKit shells out to setuid-root newuidmap/newgidmap, which needs allowPrivilegeEscalation: true with SETUID/SETGID — with NO_NEW_PRIVS set the exec fails and the pod crash-loops — plus unconfined seccomp/AppArmor profiles for user-namespace syscalls, and the native snapshotter to avoid needing /dev/fuse. It still drops all capabilities and runs as uid 1000, a long way from privileged: true, but it is the most security-sensitive manifest in the repository and its comments say so.
The pattern generalises
"Discover candidates from a system of record instead of a maintained list" applies beyond patching: rescan scheduling, SBOM freshness, even alert routing. If the registry already knows what exists and what is vulnerable, let the pipeline ask it every night.
Series: Building a Container Image Security Stack. Next: the GitOps constraints that shaped this pipeline — suspend, then run once, before trusting a schedule.