test: assert reported check status in both standard-test paths

[devin-ai-integration]

Summary

The Standard Tests had two ways for a failing check to pass a test, because the in-process and Docker paths asserted different things:

• run_test_job (in-process) asserted exactly one CONNECTION_STATUS message, and only asserted Status.FAILED inside the expect_exception() branch. A connector whose check gracefully returns AirbyteConnectionStatus(status=FAILED) (exit code 0, no trace error) passed a scenario declaring status: succeed.
• _assert_check_outcome (Docker) asserted SUCCEEDED only under expect_success(), so an ALLOW_ANY scenario (no declared status) accepted a failing check in both paths.

This extracts the assertion into airbyte_cdk/test/standard_tests/_assertions.py::assert_check_outcome and calls it from both paths, so they cannot drift again:

# _job_runner.run_test_job, verb == "check"
assert_check_outcome(
    check_result=result,
    expected_outcome=test_scenario.expected_outcome,
    connector_name=connector_root.absolute().name,
)

The assertion is also tightened, in the two ways the issue asks for:

1. ALLOW_ANY (no declared status) now requires SUCCEEDED — matching the default of CAT's ConnectionTestConfig.status (succeed). Blast radius is called out below.
2. EXPECT_EXCEPTION now requires a reported FAILED status instead of accepting the absence of any CONNECTION_STATUS message (closes gap 2 of airbytehq/airbyte-internal-issues#16212 as originally specified).

Blast radius of the ALLOW_ANY decision

Statusless configs (configs that never declare a status in any section of acceptance-test-config.yml) now have their check required to succeed. DockerConnectorTestSuite._dedup_scenarios already carries an explicitly declared status across sections, so only configs that declare no status anywhere are affected. Scanned at airbytehq/airbyte@587f1f4f706:

Connector	Statusless configs
source-s3	secrets/parquet_dataset_config.json, secrets/v4_parquet_decimal_config.json, secrets/unstructured_config.json
source-gcs	secrets/config_jsonl.json
source-gitlab	secrets/config_with_ids.json
source-google-ads	secrets/config_click_view.json, secrets/config_manager_account.json, secrets/incremental_config.json
source-mysql	integration_tests/temp/config_active.json, integration_tests/temp/config_cdc_active.json
source-shopify	secrets/config_transactions_with_user_id.json

These are all configs used for read/discovery scenarios, so their check is expected to succeed; if one does not, the new assertion surfaces a real problem rather than a false green. Still, this can turn currently-green CI red for those connectors, and the alternative (requiring an explicit status in the config) would break them immediately instead — flagging it for the reviewer to weigh in.

Caveat on the stricter EXPECT_EXCEPTION assertion

AirbyteEntrypoint.check re-raises traced exceptions whose failure_type is not config_error, emitting no CONNECTION_STATUS message at all. A scenario declaring status: failed against a connector that hard-fails that way (rather than returning Status.FAILED gracefully) will now fail the standard test. That is the behavior the issue asks for — the platform treats a non-config error as an unexpected failure, so an acceptance-test scenario should not be silently satisfied by it — but if this proves too strict across the fleet, the leniency is a one-line revert in assert_check_outcome.

Test Coverage

unit_tests/test/test_check_assertions.py (replacing unit_tests/test/test_docker_base_check_outcome.py) covers all three outcomes × (succeeded / failed / no status message), both against assert_check_outcome directly and end-to-end through run_test_job with a synthetic Source whose check reports a fixed status (or raises). The in-process cases fail without this change.

poetry run pytest unit_tests/test/ -q   # 124 passed
poetry run ruff check . && poetry run ruff format --check .

Declarative-First Evaluation

N/A — this is a CDK test-harness change, not a connector change.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17140:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17140

airbytehq/airbyte-internal-issues#16212 should be closed once this lands (its remaining gap 2 is fixed here).

Link to Devin session: https://app.devin.ai/sessions/a3d02c9188e44c789e336784dced8888
Open in Devin Desktop: https://app.devin.ai/desktop/session/a3d02c9188e44c789e336784dced8888?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1788275534-shared-check-assertion#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1788275534-shared-check-assertion

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[devin-ai-integration]

CI note on the Check: destination-motherduck job (real evidence for the blast-radius discussion above):

That job is already red on main-based PRs for credential reasons — e.g. https://github.com/airbytehq/airbyte-python-cdk/actions/runs/33446043952 and https://github.com/airbytehq/airbyte-python-cdk/actions/runs/33192062338 both fail with Your request is not authenticated. Please check your MotherDuck token and PermissionError(13, 'Permission denied'). There it fails 2 tests (the Docker-path test_docker_image_build_and_check scenarios).

On this PR the same job fails 4 tests: the same 2 plus the two in-process test_check scenarios, which previously passed despite the connector reporting Status.FAILED. That is precisely the bug this PR fixes — the in-process path was silently green on a failing check — so I don't think the extra failures indicate a regression, though it does show the change makes an already-broken credential setup fail more loudly.

Devin session

[github-actions]

PyTest Results (Fast)

4 372 tests  +9   4 360 ✅ +9   9m 34s ⏱️ -13s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 23cb5b7. ± Comparison against base commit 64bdfc9.

This pull request removes 11 and adds 20 tests. Note that renamed tests count towards both.

unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_no_status]
…

unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[allow_any_failed]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[allow_any_no_status]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[allow_any_succeeded]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[failure_failed]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[failure_no_status]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[failure_succeeded]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[success_failed]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[success_no_status]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome[success_succeeded]
unit_tests.test.test_check_assertions ‑ test_assert_check_outcome_uses_last_status[failure_last_status_wins]
…

[Copilot]

Pull request overview

Unifies and tightens how Standard Tests assert the outcome of the check command so the in-process and Docker-based paths enforce the same expectations and don’t drift.

Changes:

• Extracts shared check assertion logic into airbyte_cdk/test/standard_tests/_assertions.py::assert_check_outcome.
• Updates both Docker (docker_base) and in-process (_job_runner.run_test_job) paths to use the shared assertion.
• Replaces/updates unit tests to cover the full outcome matrix (including end-to-end via run_test_job).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
unit_tests/test/test_docker_base_check_outcome.py	Removes old unit tests tied to the Docker-only assertion helper.
unit_tests/test/test_check_assertions.py	Adds comprehensive unit tests for shared check assertion and in-process coverage.
airbyte_cdk/test/standard_tests/docker_base.py	Switches Docker check path to call the shared assert_check_outcome.
airbyte_cdk/test/standard_tests/_job_runner.py	Switches in-process check path to call the shared assert_check_outcome.
airbyte_cdk/test/standard_tests/_assertions.py	Introduces shared assertion implementation for check outcome validation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

PyTest Results (Full)

4 375 tests   4 363 ✅  12m 8s ⏱️
    1 suites     12 💤
    1 files        0 ❌

Results for commit 23cb5b7.