test(podman): run existing Podman behavioral e2e coverage in CI #3712

Activity

1. johntmyers

   added

   topic:testing

   area:compute

   on Sep 25, 2026
2. politerealism commented on Sep 27, 2026

   

   politerealism

   on Sep 27, 2026

   Contributor

   More actions

   Looking at this one
3. politerealism commented on Sep 27, 2026

   

   politerealism

   on Sep 27, 2026

   Contributor

   More actions

   Plan and order of operations

   This issue overlaps significantly with open PR #3637 ("test(e2e): run podman suite with tmachine", author elezar), which independently adds a tmachine-based e2e-podman nextest archive (43 tests passing) covering most of the 39 eligible e2e-podman targets, removes podman_userns.rs as superseded by the driver-podman tmachine suite (resolving that acceptance criterion directly), and depends on PR #3597 (migrates sync.rs coverage into conformance scenarios). Both #3637 and #3597 are currently green in CI but show mergeable: CONFLICTING and need a rebase before they can land.

   Order of operations for this work:

   1. Treat test(e2e): run podman suite with tmachine #3637 + test(conformance): migrate file transfer coverage #3597 as prerequisites. This plan does not duplicate their work and does not touch the files they modify (tests/ansible/playbooks/drivers/podman/e2e.yaml, tests/artifacts.nix) until they're on main.
   2. Track A (starting now, independent of test(e2e): run podman suite with tmachine #3637):
      • Wire podman_preflight into CI via the existing podman-external-driver-e2e job's already-exported driver binary artifact (it never runs anywhere today because that job's mise task disables the cargo-test step for other reasons).
      • Add an automated inventory/reachability check (new tasks/scripts/check-podman-e2e-coverage.sh, modeled on check-cargo-lockfiles.sh) that fails CI if a new e2e-podman-eligible test target isn't accounted for in either PODMAN_CI_TESTS or a documented exclusion — this directly satisfies the acceptance criterion preventing silent future omissions like this one.
      • Document the current test-selection contract in CI.md.
   3. Track B (after test(e2e): run podman suite with tmachine #3637 + test(conformance): migrate file transfer coverage #3597 merge):
      • Extend the coverage check to also read test(e2e): run podman suite with tmachine #3637's podmanE2eFollowUpBinaries exclusion list; file tracked follow-up issues for the exclusions that need distinct scope work (sandbox_lifecycle conformance migration, podman_oci_identity SPIFFE fixture plumbing, provider_refresh_handles's own purpose-built suite), and link transparent_tcp's exclusion to the existing test(e2e): restore musl DNS probe coverage for guest prebuilt artifacts #3009 rather than duplicating it.
      • Investigate the remaining plausible quick-fix targets (provider_auto_create, proxy_egress_pipeline, websocket_conformance, workspace_lifecycle) against a live tmachine environment.
      • Add a rootful leg to the new tmachine e2e-podman testsuite (its current playbook hardcodes rootless-only values instead of using the tmachine_container_runtime role the way driver-podman's playbooks already do).
      • Wire driver-podman/e2e-podman into Release Dev and Release Tag qualification, mirroring the existing provider-refresh job pattern.
      • Add visible selected-vs-eligible test-count reporting so a narrow run can't be mistaken for full coverage.

   PRs for Track A will follow this comment. Track B PRs will follow once #3637/#3597 land.
4. politerealism commented on Sep 29, 2026

   

   politerealism

   on Sep 29, 2026

   Contributor

   More actions

   Status update given everything that's landed since the plan above:

   • podman_preflight is resolved via test(podman): move podman_preflight into driver-podman integration tests #3783 (reopened and approved by @elezar, /ok-to-test applied, pending merge) rather than the original test(podman): wire podman_preflight into CI and add a Podman e2e coverage check #3749 approach.
   • podman_userns is resolved via test(e2e): run podman suite with tmachine #3637 (superseded by the driver-podman tmachine suite).
   • podman_resource_limits now runs for real against a live gateway via test(e2e): run podman suite with tmachine #3637's tmachine e2e-podman archive.

   @elezar — given #3460 (Nix/tmachine target-state doc) and the string of test(conformance)/test(tmachine) PRs you've got in flight (#3766, #3768, #3792, #3795, #3839, etc.), it looks like most of the remaining shared-target and driver-specific coverage gaps here (the 6 of 12 shared targets still excluded, podman_oci_identity, provider_refresh_handles) will fall out of that migration rather than needing separate Podman-specific CI wiring. To avoid duplicating effort, I'm deliberately not touching that migration.

   What I'll pick up next instead, since it's orthogonal to the conformance/driver-specific split:

   • Release Dev and Release Tag bundled Podman behavioral qualification (currently absent from both).
   • Selected-vs-eligible test-count reporting in job names/summaries, so a narrow run can't be mistaken for full coverage.
   • Documenting the rootless-only scope of the current e2e-podman tmachine archive (only fedora-podman-rootless runs it; podman_resource_limits specifically would benefit from a rootful leg since it reads real cgroup v2 state and rootless cgroup delegation is the harder/riskier case).

   Flag anything above that's already in flight or planned elsewhere so we don't collide again.
5. elezar commented on Sep 30, 2026

   

   elezar

   on Sep 30, 2026

   Member

   More actions

   Thanks @politerealism. I am busy constructing a plan to migrate the e2e/rust tests that are covered by the e2e-podman feature so that we can tackle that in parallel.

   👍React with 👍2Reacted by Polite_realism and Matthew Grossman
6. politerealism

   added a commit that references this issue on Oct 3, 2026

   docs(podman): document and track the e2e-podman rootful coverage gap

   ...

   6015c8e
7. politerealism commented on Oct 6, 2026

   

   politerealism

   on Oct 6, 2026

   Contributor

   More actions

   Update 2026-10-06: Following up on the 2026-09-29 note above where I deferred the e2e/rust→conformance migration to @elezar to avoid duplicating his in-flight work.

   Coordinated directly with Evan today on the 11 general-conformance candidates I'd flagged in a research pass on #3460. He confirmed his own plan (verify under Podman, relax gates on provider_refresh_handles.rs/websocket_conformance.rs, sandbox_labels.rs needs nothing) and suggested migrating applicable ones straight into tests/suites/conformance/ rather than gate-relaxing in place — and invited a partial migration on my end rather than a hard handoff.

   Migrated so far (branch not yet opened as a PR, pending final review on my end):

   • workspace_lifecycle.rs → workspace-lifecycle/workspace-terminating scenarios
   • upload_create.rs → file-transfer/create-upload scenario (grouped with the existing file-transfer family)
   • provider_auto_create.rs → provider-auto-create scenario
   • sandbox_templates.rs → 4 scenarios (lifecycle, get-after-delete, duplicate-name, missing-template)
   • settings_management.rs → settings-management scenario

   All verified against an ephemeral Podman gateway. Will reference #3954 (not this issue) when the PR opens, since that's the umbrella tracking issue for the RFC-0016 testing-strategy work this falls under.

   Two things surfaced along the way worth noting here:

   • port_forward.rs doesn't fit the conformance harness's CLI request/response model (needs a long-lived background SSH-tunnel process + raw socket I/O) — holding that one for Evan's input on whether that pattern belongs on OpenShellRunner or as a scenario-local bypass.
   • Found two of Evan's already-landed file_transfer.rs conformance scenarios (round-trip, git-filtering) currently failing against main — pre-existing, unrelated to this migration, flagged to him separately.