Repository navigation
ci(apko-plan,smoke): run the smoke test on a runner with a docker daemon - #62
Merged
raffaele-oplabs merged 1 commit intoSep 11, 2026
Merged
Conversation
The default amd64 smoke runner was `ubuntu-slim`, which has no docker daemon. `docker run` there writes "failed to connect to the docker API" into the output the loop captures and exits 1, which is not in the narrow set of exit codes the loop treats as fatal — so the job passed without the image ever starting. Every amd64 smoke leg in the consumers that do not override `smoke_runners` has been green on a test that never ran. It surfaced only on a release tag, where `expected_version` had no output to match and failed with "no smoke command reported expected version". Two changes, because either alone leaves the hole open: - default both arches to a runner that has a daemon, and - pull the image before the loop, so a runner that cannot fetch or cannot execute it fails the job with a clear message instead of falling through to a version assertion that may not even be enabled. The arch check matters for the same reason as the pull: `docker pull --platform` will fetch a foreign arch happily, and the `exec format error` that follows is another tolerated exit code. The planner's default was untested — the fixture set `smoke_runners` explicitly, so only the override path was covered. Both defaults and the partial-override fallback are asserted now; each assertion was mutation-checked against the code it covers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
raffaele-oplabs
approved these changes
Sep 11, 2026
raffaele-oplabs
left a comment
Contributor
There was a problem hiding this comment.
Reviewed and it seems there was an issue in the initial setup. LGTM!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The smoke tests have never run
The default amd64 smoke runner is
ubuntu-slim, which has no docker daemon. On that runnerdocker runwritesinto the output the loop captures, and exits 1 — which is not in the
125/126/127/129-159set the loop treats as "could not run". So the step falls through to the version assertion, and whenexpected_versionis empty (every non-release build) the job passes green having never started the image.Confirmed in the logs of runs that reported success:
ufmandop-conductor-opsamd64 legsIt surfaces as a failure only on a release tag, where
expected_versionis set and has nothing to match. That is the current state ofop-reth-premium/v0.0.2-rc.2: melange and apko succeeded, thenFAIL: no smoke command reported expected version v0.0.2-rc.2.Affected: every amd64 smoke leg in
optimism-premium,infraandinfrastructure-services.optimismis unaffected — it pins both arches insmoke_runnersexplicitly, which is what let the bad default survive this long.ubuntu-slimappears nowhere else in this repo.What changes
The default gets a daemon. amd64 defaults to
ubuntu-24.04, matching whatoptimismhad already worked around to. arm64 was alreadyubuntu-24.04-armand is unchanged.The image is fetched before the loop. A runner that cannot pull now fails the job with docker's own error, rather than handing the loop an error string to not-match a version against. This is the half that keeps the hole shut if a future override reintroduces a runner without a daemon — the default alone leaves the same silent pass one config edit away.
The pulled arch is checked.
docker pull --platformfetches a foreign arch happily, and theexec format errorthat follows is exit 126 or 1 depending on docker version and image store; exit 1 is tolerated by the loop. Reachable today only by writing a mismatchedsmoke_runnersoverride — plausible as a way to dodge arm runner cost — so it is checked rather than left to the arch defaults happening to line up.Testing
The planner's smoke-runner default was untested: the fixture config sets
smoke_runnersexplicitly, so only the override path was covered. Added coverage for both defaults and for the partial-override fallback (an override naming one arch must not capture the other).Each new assertion was mutation-checked against the code it covers:
ubuntu-slimexpected 'ubuntu-24.04', got 'ubuntu-slim'smoke_runnerdeletedexpected 'ubuntu-24.04-arm', got 'ubuntu-24.04'smoke_runnerslookup ignoredexpected 'ubuntu-latest', got 'ubuntu-24.04'apko-plan.test.shpasses;shellcheck actions/apko-plan/*.shclean; both touched workflows lint clean.Not verifiable locally: the
docker pullpreflight and the arch check run only on a GitHub runner.Cost
One extra
docker pullper smoke leg, on an image the loop was about to pull implicitly anyway. The amd64 default moves fromubuntu-slimtoubuntu-24.04— a fraction of a runner-minute per leg, in exchange for the test actually running.🤖 Generated with Claude Code
Consumer side: optimism-premium#267 (interim
smoke_runnersoverride, unblocks the blocked release now) and optimism-premium#268 (pins this and reports the upstream version on the apko path).Blast radius: which smoke tests start running for real
Flipping the default turns previously-no-op amd64 legs into real tests, so it is worth knowing which consumers could newly go red.
Every image with a
smoke_testininfra(10) andinfrastructure-services(9) inherits the top-level"apko_archs": "amd64,arm64". Their arm64 legs already run onubuntu-24.04-arm, which has a daemon — so the identical--help/--versioncommand is already validated for real on every one of those images, and the amd64 leg becoming real should not newly break any of them.The exception is
op-reth-premium, which sets"apko_archs": "amd64". It has no arm64 leg, so its smoke command has never executed anywhere. optimism-premium#267 is the first run of it, and is the one place this PR could surface a genuine pre-existing problem rather than just start enforcing a passing test.