Your Most Secure Images Are the Ones Copacetic Can Never Patch

Here is a paradox worth internalising: the images you adopt because they are more secure are precisely the ones automated OS patching cannot touch.

The observation

The gap, drawn: the patcher's mechanism ends where the hardened image's design begins.

Our nightly Copacetic job observed this live. n8nio/n8n:2.23.3 is built on a Docker Hardened Image (Alpine Linux v3.22) and reports 96 fixable OS CVEs. Copa cannot patch a single one, because apk is not present in the image.

This is not a Copacetic defect. Distroless, Chainguard, Wolfi, and Docker Hardened Images deliberately strip the package manager. Smaller attack surface, no shell to poke, fewer moving parts — all true and all good. But Copa's mechanism is "run the image's package manager via BuildKit to apply published updates". Remove the package manager and there is nothing for it to run.

The subtle part: Trivy still reports fixable CVEs, because the package database survives even when the tooling does not. So your scan results insist the image is patchable while your patcher insists it is not. Both are telling the truth about different things.

What we changed

The nightly job classifies these images as unpatchable rather than failed, and reports the list separately at the end of each run. The distinction matters operationally:

• A failure means the pipeline is broken and someone should look.
• An unpatchable image means the pipeline works fine and the base image needs a bump — which is Renovate's job, not Copacetic's.

If an image shows up in the unpatchable list night after night, that is a signal about the base image's vendor: either they publish fixed rebuilds (bump and get the fixes) or they do not (choose a different base). Either way, the answer is never "debug Copacetic".

The wider lesson

"Automated patching" always resolves to a two-track answer:

1. Copacetic track — traditional distro bases with a package manager. Fast, no rebuild, done tonight.
2. Rebuild track — distroless, hardened, and all application dependencies. Renovate MR, CI tests, merge.

Choosing a hardened base moves an image from track 1 to track 2 permanently. That is usually the right trade — you are trading a nightly automated patch for a smaller surface and a vendor who ships fixed rebuilds — but it should be a known trade, not a surprise discovered during an incident.

Checklist before you blame the patcher

When an image reports fixable CVEs but Copacetic reports nothing to do:

1. Does the image contain apt-get, apk, yum, or microdnf?
2. Is the base EOL? (Archived apt repos mean even Copa's package-manager route has nothing to fetch. trivy image --exit-on-eol 1 catches this.)
3. Is it multi-arch with a non-amd64 platform you need? Copa's rootless BuildKit worker in our deployment is amd64-only; other platforms in an index pass through unpatched, by design.

If all three are yes/no in the wrong direction, the fix is a base bump or a rebuild — and that finding belongs in the Renovate backlog, not the Copacetic logs.

Series: Building a Container Image Security Stack. Next: how the nightly patcher discovers its own targets from Harbor's scan results — no list to maintain, and why that matters more than it sounds.