Repository navigation
ci: separate e2e test families from driver target variants #4540
Description
User Story
As an OpenShell maintainer reorganising the project's test lanes for RFC-0016, I need conformance, feature-specific, and driver-specific execution to be identifiable separately, while preserving the tests and target combinations that currently protect changes and releases.
Description
Implement the first orchestration migration to three e2e families. Reuse the current integration-conformance and integration-feature suites. Collect existing independent runtime lanes under driver-specific execution and separate their embedded conformance execution into the conformance family.
Treat bundled versus external compute drivers as target variants across applicable families. Preserve the existing external VM runtime tests as well as its conformance tests. The other external driver lanes currently run conformance only; expanding those runtime suites is follow-up work.
Retain current preparation mechanisms, target matrices, label selections, required/optional/advisory policy, and release qualification profile. Keep existing combined local commands compatible. Mixed legacy runtime suites can remain in the driver family pending assertion-level extraction.
Context
This is a focused implementation under the testing-strategy tracker #3954 and RFC-0016 discussion #3460. Closing this issue does not complete the overall strategy, suite-source migration, broader external-driver coverage, or broader release qualification.
The shared Cargo conformance package already replaced the standalone conformance executable in #3866. Reuse that package rather than introducing another runner.
Definition of Done
- Branch and release workflows expose conformance, features, and drivers families using shared runtime preparation.
- Each driver harness can run conformance and its existing runtime suites separately, preserving combined local execution by default.
- Existing test selectors, installers, architectures, driver variants, Kubernetes versions/modes, GPU targets, and advisory/optional distinctions retain an explicit execution path.
- The external VM target runs both families; external Docker, Podman, and Kubernetes retain their existing conformance coverage.
- A selected job that fails, is cancelled, is skipped, or has no result cannot yield a successful family or release qualification result.
- Focused regression checks cover selection, preserved matrices, suite dispatch, and failure propagation.
- Testing documentation and CI diagnostics guidance explain the implemented structure and remaining mixed suites.
- Live CI evidence is reviewed before the replacement lanes are merged; coverage gaps remain visible.
Activity