Skip to content

ci(apko-plan,smoke): run the smoke test on a runner with a docker daemon - #62

Merged
raffaele-oplabs merged 1 commit into
ethereum-optimism:mainfrom
geoknee:fix/apko-smoke-docker
Sep 11, 2026
Merged

raffaele-oplabs merged 1 commit into
ethereum-optimism:mainfrom
geoknee:fix/apko-smoke-docker

Conversation

@geoknee

@geoknee geoknee commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

The smoke tests have never run

The default amd64 smoke runner is ubuntu-slim, which has no docker daemon. On that runner docker run writes

failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running

into the output the loop captures, and exits 1 — which is not in the 125/126/127/129-159 set the loop treats as "could not run". So the step falls through to the version assertion, and when expected_version is empty (every non-release build) the job passes green having never started the image.

Confirmed in the logs of runs that reported success:

It surfaces as a failure only on a release tag, where expected_version is set and has nothing to match. That is the current state of op-reth-premium/v0.0.2-rc.2: melange and apko succeeded, then FAIL: no smoke command reported expected version v0.0.2-rc.2.

Affected: every amd64 smoke leg in optimism-premium, infra and infrastructure-services. optimism is unaffected — it pins both arches in smoke_runners explicitly, which is what let the bad default survive this long. ubuntu-slim appears nowhere else in this repo.

What changes

The default gets a daemon. amd64 defaults to ubuntu-24.04, matching what optimism had already worked around to. arm64 was already ubuntu-24.04-arm and is unchanged.

The image is fetched before the loop. A runner that cannot pull now fails the job with docker's own error, rather than handing the loop an error string to not-match a version against. This is the half that keeps the hole shut if a future override reintroduces a runner without a daemon — the default alone leaves the same silent pass one config edit away.

The pulled arch is checked. docker pull --platform fetches a foreign arch happily, and the exec format error that follows is exit 126 or 1 depending on docker version and image store; exit 1 is tolerated by the loop. Reachable today only by writing a mismatched smoke_runners override — plausible as a way to dodge arm runner cost — so it is checked rather than left to the arch defaults happening to line up.

Testing

The planner's smoke-runner default was untested: the fixture config sets smoke_runners explicitly, so only the override path was covered. Added coverage for both defaults and for the partial-override fallback (an override naming one arch must not capture the other).

Each new assertion was mutation-checked against the code it covers:

mutation caught by
amd64 default back to ubuntu-slim expected 'ubuntu-24.04', got 'ubuntu-slim'
arm64 branch of smoke_runner deleted expected 'ubuntu-24.04-arm', got 'ubuntu-24.04'
smoke_runners lookup ignored expected 'ubuntu-latest', got 'ubuntu-24.04'

apko-plan.test.sh passes; shellcheck actions/apko-plan/*.sh clean; both touched workflows lint clean.

Not verifiable locally: the docker pull preflight and the arch check run only on a GitHub runner.

Cost

One extra docker pull per smoke leg, on an image the loop was about to pull implicitly anyway. The amd64 default moves from ubuntu-slim to ubuntu-24.04 — a fraction of a runner-minute per leg, in exchange for the test actually running.

🤖 Generated with Claude Code


Consumer side: optimism-premium#267 (interim smoke_runners override, unblocks the blocked release now) and optimism-premium#268 (pins this and reports the upstream version on the apko path).

Blast radius: which smoke tests start running for real

Flipping the default turns previously-no-op amd64 legs into real tests, so it is worth knowing which consumers could newly go red.

Every image with a smoke_test in infra (10) and infrastructure-services (9) inherits the top-level "apko_archs": "amd64,arm64". Their arm64 legs already run on ubuntu-24.04-arm, which has a daemon — so the identical --help/--version command is already validated for real on every one of those images, and the amd64 leg becoming real should not newly break any of them.

The exception is op-reth-premium, which sets "apko_archs": "amd64". It has no arm64 leg, so its smoke command has never executed anywhere. optimism-premium#267 is the first run of it, and is the one place this PR could surface a genuine pre-existing problem rather than just start enforcing a passing test.

The default amd64 smoke runner was `ubuntu-slim`, which has no docker
daemon. `docker run` there writes "failed to connect to the docker API"
into the output the loop captures and exits 1, which is not in the
narrow set of exit codes the loop treats as fatal — so the job passed
without the image ever starting.

Every amd64 smoke leg in the consumers that do not override
`smoke_runners` has been green on a test that never ran. It surfaced
only on a release tag, where `expected_version` had no output to match
and failed with "no smoke command reported expected version".

Two changes, because either alone leaves the hole open:

- default both arches to a runner that has a daemon, and
- pull the image before the loop, so a runner that cannot fetch or
  cannot execute it fails the job with a clear message instead of
  falling through to a version assertion that may not even be enabled.

The arch check matters for the same reason as the pull: `docker pull
--platform` will fetch a foreign arch happily, and the `exec format
error` that follows is another tolerated exit code.

The planner's default was untested — the fixture set `smoke_runners`
explicitly, so only the override path was covered. Both defaults and
the partial-override fallback are asserted now; each assertion was
mutation-checked against the code it covers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@geoknee
geoknee requested review from a team as code owners September 10, 2026 09:33

@raffaele-oplabs raffaele-oplabs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and it seems there was an issue in the initial setup. LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants