Skip to content

ci: separate e2e test families from driver target variants #4540

Closed

Description

@elezar

User Story

As an OpenShell maintainer reorganising the project's test lanes for RFC-0016, I need conformance, feature-specific, and driver-specific execution to be identifiable separately, while preserving the tests and target combinations that currently protect changes and releases.

Description

Implement the first orchestration migration to three e2e families. Reuse the current integration-conformance and integration-feature suites. Collect existing independent runtime lanes under driver-specific execution and separate their embedded conformance execution into the conformance family.

Treat bundled versus external compute drivers as target variants across applicable families. Preserve the existing external VM runtime tests as well as its conformance tests. The other external driver lanes currently run conformance only; expanding those runtime suites is follow-up work.

Retain current preparation mechanisms, target matrices, label selections, required/optional/advisory policy, and release qualification profile. Keep existing combined local commands compatible. Mixed legacy runtime suites can remain in the driver family pending assertion-level extraction.

Context

This is a focused implementation under the testing-strategy tracker #3954 and RFC-0016 discussion #3460. Closing this issue does not complete the overall strategy, suite-source migration, broader external-driver coverage, or broader release qualification.

The shared Cargo conformance package already replaced the standalone conformance executable in #3866. Reuse that package rather than introducing another runner.

Definition of Done

  • Branch and release workflows expose conformance, features, and drivers families using shared runtime preparation.
  • Each driver harness can run conformance and its existing runtime suites separately, preserving combined local execution by default.
  • Existing test selectors, installers, architectures, driver variants, Kubernetes versions/modes, GPU targets, and advisory/optional distinctions retain an explicit execution path.
  • The external VM target runs both families; external Docker, Podman, and Kubernetes retain their existing conformance coverage.
  • A selected job that fails, is cancelled, is skipped, or has no result cannot yield a successful family or release qualification result.
  • Focused regression checks cover selection, preserved matrices, suite dispatch, and failure propagation.
  • Testing documentation and CI diagnostics guidance explain the implemented structure and remaining mixed suites.
  • Live CI evidence is reviewed before the replacement lanes are merged; coverage gaps remain visible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions