Skip to content

Harden the fork CI reconciler and protect the fork CI gate - #74909

Open
potiuk wants to merge 16 commits into
fork-ci-5-docsfrom
fork-ci-6-reconciler-followups
Open

potiuk wants to merge 16 commits into
fork-ci-5-docsfrom
fork-ci-6-reconciler-followups

Conversation

@potiuk

@potiuk potiuk commented Oct 17, 2026

Copy link
Copy Markdown
Member

Stacked on #74825. Part 6 of the AIP-120 implementation (design: #73378); this PR's base is the previous branch, so the diff shows only this layer.

Follow-ups to the reconciler in #74822, one commit each, plus protection of the gate itself:

  • A "push again" nudge is posted before the state comment records awaiting_push, so a failed post is retried.
  • A workflow or trigger the base branch does not have counts as project CI in the tampering check only if one of its jobs ran.
  • Pull requests sharing a head commit across different bases stay green_unverified, since one commit status cannot hold a decision per base.
  • Recorded runs made while the pull request targeted a base without the gate are ignored; a run that could belong to another pull request from the same branch is reported to Slack instead of closing.
  • A gated pull request that changes the fork CI gate (gate code, configuration beyond the author's own [pilot] entry, the gate job or its wiring in ci-amd.yml, or the monitor) is closed with a comment explaining why. Each of those files says so. The state comment's marker records gate_checked only when this check ran for the head commit, so pull requests reconciled before it existed are checked once.

closes: #74905
closes: #74906
closes: #74907
closes: #74908


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

AIP-120 moves CI for external contributions into contributor forks so
project runners serve committers, collaborators and exempted
organisations, and so maintainers only review pull requests that
already passed in the author's fork. The mechanics span the main test
workflow, a scheduled reconciler for gated pull requests, a commit
status that surfaces the fork CI state with a link to the fork run, a
local pre-flight command and the contribution docs, so the design is
recorded before implementation and the decisions that shape it are
captured as ADR 0018.

Generated-by: Claude Opus 5
The gate job in ci-amd.yml runs from the pull request's merge commit,
so a gated external pull request can drop the gate or add a workflow
with a pull_request trigger and get project CI anyway. The design
accepted that as a one-run cost but left it unbounded per pull
request, because every further push would get another run.

The reconciler now detects this after the fact from what GitHub
recorded rather than from the pull request tree: project CI ran for
the head SHA and the pull request changes files under
.github/workflows/. Both signals are required so that legitimate
project CI runs (backlog reset, a removed "use project ci") and
legitimate workflow edits that keep the gate intact are never closed.
Closing stops further pull_request runs, so one attempt buys one run,
and the closure is reported to Slack as an attention item rather than
as a reconciler failure. ADR 0018 records the trade-off, including the
external pull requests with wanted workflow changes that must keep
"use project ci".

Generated-by: Claude Opus 5
The dev list agreed to start with an opt-in pilot by a few volunteer
contributors rather than enabling fork CI for everyone at once. The
committer-only "use fork ci" label cannot serve that: it arrives after
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJq0YVxCRC1aQ7uu5UhlAAAcYgQACYDDrIdXBZxVhqczF3TNrO7
 qdkPgQy3NgLfVgmY0iy7ypzDRYY+glMD415Qc5OJy7n5M7XBqJ3ALQf04U/IGm0O
 bMXgq1+Z9B/sRz27VNQYJm9X30hL+pE4iYHwGAQHXsr/TIgl2WN+V4awsRYEVQXM
 x3Q/jwR12/YbiW2R3c61J+c7mrLdw4bZ6UtnDQqWUYkimgRGDLFvlZ/yMAj8kBuU
 UyXGeOtTaIJHlghczgxlTeBDNMjuE9o5S6xsW5n+eWJXy7E/883L6uMWR3OjLR01
 l2gpSqrpi52xYMVd8Zik+gpuWjkQ45C9k+XgGQfLfhoelZJ8uZfToMqARne4xVlZ
 4hFIHfZSGdcLCi+sbU6PKxrMdX7HF4vDAxP9Tv4IpIzANRxG9IH5uXpHwAAx24LF
 ogZQDyykjqTsadl7SqjKDw/Hybv4hklnnr8BFfvYd54iFkkP3fqSizlV4ecr1tyJ
 OBvI7DbNNNdySebmnMYQGBziLQF5uUseD8srhCiXG5XNaSqbOnNXc+FHZmn0oU3O
 EJ15O0QPi4rkRXQ5gYCjjb4tO99H3Mw6bMgiIWTjVa/1QPmOb9mtEbpRP/ESHuOe
 1VgYd8tUczg6FB6Po5f/DXV2rZB36ij9Bo4nBmR7V1HApOx4E88lDIznGpqP/cWf
 MetUk2Nom7/sr+UaBu3i
 =wncY
 -----END PGP SIGNATURE-----
 
the first pull_request run has already used project CI, and a
maintainer would have to label every pull request the volunteer opens.
A per-login join instant gates all their new pull requests from the
first run while leaving older ones on project CI, so those can never be
mistaken for gate tampering.

The reconciler also has to run while the gate is disabled, which the
earlier "exits early" wording contradicted for the label pilot as well.

Generated-by: Claude Opus 5
Enabling ci-amd.yml in a fork also enables its cron, which would run the
full canary matrix in every contributor fork twice a day. A join instant
or enabled_since earlier than the merge that sets it gates pull requests
that already ran project CI. Forks not named "airflow" would never show
a fork run, and a pull request reopened after an inactivity close would
be closed again at once. ADR 0018 was already taken.

Generated-by: Claude Opus 5
A fork run executes the workflows, breeze and CI scripts of the pushed commit, so a pull request changing them decides its own fork CI result. Gate tampering was only checked on the head commit, so pushing a tampered commit and then a clean one hid the bypass, and the check could be dodged by renaming a job to the gate's name or by truncated check-suite listings. The gate also fell back to project CI when its label read failed, which an exhausted token quota could trigger for every pull request.

Generated-by: Claude Opus 5
The gate job and the reconciler must agree on which pull requests use project CI and which are left to fork CI, so the decision lives in one standard-library-only function that the gate can run without installing breeze, driven by one reviewed configuration file. It ships disabled.

Generated-by: Claude Opus 5
Contributors must get CI from a plain push to their fork, and project
CI must not run for pull requests left to fork CI, without reacting to
pull request events with write permissions. A gate job ahead of
build-info decides from the base branch's configuration, and the push
trigger widens to every branch while the gate keeps the old branch list
in effect for apache/airflow.

Skipping build-info also skips everything that depends on it, which the
always() conditions of finalize-tests and notify-slack would otherwise
treat as success. Pilot join instants are compared with pull request
creation but only take effect on merge, so the config hook keeps new
ones in the future.

Generated-by: Claude Opus 5
A push in a contributor fork would otherwise be treated as a post-merge push and run the full matrix, about 220 minutes on the contributor's 20-runner cap, for every push. Computing the change set against apache/airflow gives the same selection a pull request would get, independent of how stale the fork's main is, and fork runs must never select self-hosted runners that forks do not have.

Generated-by: Claude Opus 5

@muditagrawal2007 muditagrawal2007 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stacked-on-#74825 design is clean, and the "post the closing comment before the state comment records the new state" change is the right call — a failed post was silently lost before this PR. The gate-protection added layer is also a sensible guardrail for the same class of self-deciding risk that motivated the original gate. Read the full diff against fork-ci-5-docs; below is what stood out.

1. The gate-modified marker doesn't record gate_checked: true

The closure paths for gate_bypassed, closed (inactivity) and the regular state branches all now write Marker(..., gate_checked=gate_checked) — but the new gate_modified branch in release/classify writes Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now) without that flag:

_set_marker_comment(
    plan,
    pr,
    state_comment(
        Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now), fork=pr.head_repo, run_url=None
    ),
)

The documentation states the opposite:

The state comment's marker records "gate_checked": true with the head commit only when this check ran for it

The check did run for this head commit — that's why we're in this branch. So the comment and the doc disagree.

The reason this matters in practice is maintainer-reopen-without-restore. Reading the docstring for reopened_since_closing, when marker.state == gate_modified and pr.last_reopened_at > marker.since, the "reopen since closing" path is skipped (correctly), and execution falls through to the gate-check block. With gate_checked: false on the marker, the check re-runs and closes the PR again — which matches the documented intent:

a maintainer's reopen without restoring the gate closes it again on the next run

So either:

  • The code is right and the doc sentence above is overstated (e.g., add an exception for "and the close path retains gate_checked: false so a reopen re-runs the check"), or
  • The code should record gate_checked: true and already_checked should additionally require pr.last_reopened_at is None or pr.last_reopened_at <= marker.since. That keeps the once-per-head-commit guarantee and still re-runs after a maintainer reopen.

The second option is more consistent with the rest of the diff and with the rule "checked once per head commit". Right now the rule is held together by the same exception that makes the doc misleading.

already_checked = (
    marker is not None
    and marker.gate_checked
    and marker.sha == pr.head_sha
    and (pr.last_base_change_at is None or pr.last_base_change_at < marker.since)
    and (pr.last_reopened_at is None or pr.last_reopened_at <= marker.since)
)

Then record gate_checked=gate_checked on the gate-modified marker too, and add a test for the reopen-without-restore case (test_gate_change_reopens_closes_again or similar).

2. Gate-modified plans don't ask for the gate_checked sentinel that other closures carry

Same point as #1, expressed from the consumer side. test_a_change_to_the_gate_closes only asserts '"state": "gate_modified"' in plan.marker_comment, which passes whether or not the gate-checked flag is set. Tightening that to "gate_checked": true would lock in whichever resolution you pick and surface regressions in either direction. The other three tests for this branch (test_unfinished_gate_change_closure_is_retried, test_marker_records_whether_the_gate_was_checked, test_gate_change_is_checked_once_per_head_commit) sidestep the close path.

3. classify is now ~270 lines doing everything

Counting from the start of classify to its return plan, the function decides: which paths skip the gate (use project ci/use fork ci), gate tampering, gate-modified closure, "reopen since closing", fork state observation, green-trust decision (unverified vs verified), the shared-head-across-bases path, draft/labels, status, marker. None of these are individually complicated but reading them as one diff is. A small refactor — extracting _plan_closure(state, comment_factory, *, change=...) for the three close paths and _plan_record_state(state, since) for the record-only paths, with classify as a top-level orchestrator — would make the diff much easier to review and the existing tests can stay because they assert on Plan shape.

4. observe_fork_state doesn't know about gate_modified

def observe_fork_state(fork: ForkObservation, marker: Marker | None) -> tuple[ForkState | None, str | None]:
    ...
    if fork.workflow_state is None:
        if marker is not None and marker.state == ForkState.AWAITING_PUSH.value:
            return ForkState.AWAITING_PUSH, None
        return None, "no fork run and the fork workflow state was not read"

For a re-opened gate_modified PR with no fork run, this returns (None, "no fork run...") rather than (GATE_MODIFIED, None). That's tolerable because the release/gate-check block runs before observe_fork_state and either closes again or falls through (depending on #1), but if you do end up extending the gate-check block to be re-entrant from observe_fork_state, the function needs an explicit gate_modified case, otherwise the post-#1 architecture risks an unknown line per run.

5. _other_pull_requests is re-issued per run, not cached per branch

BypassFinder._other_pull_requests calls rest_get(pulls?head=...) with pagination. The result depends only on (head_repo, head_ref, window_start) and find() is invoked once per branch by run_reconcile. Hoisting it into BypassFinder.__init__ (computed once with the same 6h window) would save one paginated call per PR per run. Trivial cost per call, but the call pattern repeats across every reconciles and shows up in the GitHub calls: {...} log line.

6. _set_status URL is hardcoded in the new branch

_set_status(
    plan, pr, ForkState.GATE_MODIFIED.value, f"https://github.com/{repo}/pull/{pr.number}/files"
)

vs. the helper used elsewhere — fork.run_url or fork_actions_url(pr.head_repo) and f"https://github.com/{repo}/pull/{pr.number}/checks". The hardcoded value is fine semantically (a gate_modified PR has no fork run to link to; linking the changed files is the right thing), but the inconsistency is worth a helper for consistency — something like gate_changed_files_url(pr.number) if you don't want to inline it.

7. _list_jobs raises and goes through the except GitHubError path

try:
    jobs = self._list_jobs(run)
except GitHubError:
    result = True
else:
    result = ...

That result = True on the right-hand side of the OR-test for "did work" reads the wrong way at first glance — "no jobs listed → counts as project CI" is exactly the intended semantic, but the previous shape made the same point by setting result = True before checking anything and the diff moved it under a guard. Worth a two-line comment at the assignment:

except GitHubError:
    # A listing we can't make fails open: better to flag a run that turned out
    # to be project CI than to miss one.
    result = True

Two of the parametrized tests cover the False branch; there isn't a test that exercises rest_get raising on the jobs call. Adding one would lock in the "fail open on listing error" semantics.

8. base_at boundary semantics

for changed_at, previous, _ in reversed(pr.base_changes):
    if when < changed_at:
        base = previous

For a run whose created_at exactly equals a BaseRefChangedEvent.createdAt (when == changed_at), the new base applies. That's right for a run triggered by the retarget itself (pull_request with edited: change_base), which sees the new base. For a run whose created_at is one millisecond less than the retarget (when < changed_at), the loop sets base = previous (old base). That's also defensible — the run was triggered against the old base. The asymmetry around the boundary is intentional but worth a one-line comment; "runs at or after a retarget are considered to be against the new base".

9. Schema test for MONITORED_GATE_PROTECTED_PATHS?

GATE_PROTECTED_PATHS is a single frozenset at module scope; if a future PR renames a file or adds another gate piece (e.g., fork_ci_maintainer.py) the maintainer who lands it needs to remember to add it both here and to the four "this file is part of the gate" comments. A small assert that mirrors the comments — pulling the strings out of those four header comments and checking every protected path has a back-reference — would catch a forgotten update. Optional, but it's the kind of doc/code drift the existing comments are trying to prevent.

10. Smaller things

  • Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now) is built both inline (state_comment call) and via a future helper — at this PR's size, the duplication is small but a const for the per-state string would help.
  • from dataclasses import replace — is_own_pilot_change imports it but doesn't use replace directly... wait, it does: return replace(head, pilot=base.pilot) == base and .... Fine.
  • PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n" lives at module scope in the test file. Minor — fine for tests, but if anyone wants a second file to reference "what does an unchanged base look like" they'll copy-paste.
  • The "Gate protection" section in 08_fork_ci_for_external_contributors.md is excellent and the table-row addition plus the **New comments go first.** rule are real, easily-understood improvements.

Overall the diff lines up with its stated design, the test density matches the risk of each new branch (≥6 cases for "checked once per head commit" alone), and the gate tampering and shared-head paths are clearly motivated by #74905 / #74906 / #74907. The one thing I'd want resolved before merge is the gate-modified marker flag conflict in #1/#2 — whichever way you settle it, it's worth a one-line comment where the marker is built explaining why gate_checked is on every other closure's marker and the exception.

Gated pull requests need labels, draft state, a state comment, a commit
status that links to the fork run, and closing after inactivity or gate
tampering, and none of that may be written in reaction to an untrusted
pull_request event. A scheduled, idempotent reconciler that derives every
change from what GitHub recorded is the only writer, so missed or
repeated runs are harmless and nothing runs pull request code with write
permissions. Whatever it cannot classify is reported to maintainers
instead of being guessed.

The monitor needs the breeze setup action, which frees disk space with
sudo and docker, so it runs on ubuntu-latest rather than ubuntu-slim.

Generated-by: Claude Opus 5
Fork CI only works when the contributor's remotes, fork and fork workflow are set up the way the gate and the reconciler expect, and committers must keep fork CI off so their pushes do not run twice. A read-only audit lets contributors verify all of that before pushing and offers the exact fix for anything missing. Remote parsing moves out of ci upgrade so both commands agree on what a correct setup is, and it now matches apache/airflow exactly instead of any URL containing it.

Generated-by: Claude Opus 5
Contributors need to know how to set up CI in their fork, how to check that setup, and how to join or leave the pilot, and maintainers need to know which parts of a gated pull request's lifecycle the reconciler owns so triage does not fight it. The previous one-line advice to enable GitHub Actions in the fork did not say any of that.

Generated-by: Claude Opus 5
The state comment is what the next reconciler run compares with. Recording
awaiting_push there before the nudge was posted meant a failed post was never
retried, and the contributor got no prompt before the inactivity closure.

Generated-by: Claude Opus 5
…ering check

A workflow or trigger the base branch does not have always counted as
project CI, so a pull request adding a fork-only workflow that its jobs
guard off in apache/airflow was closed for gate tampering although nothing
ran.

Generated-by: Claude Opus 5
…aintainer

The green-trust decision is stored as a commit status, which every pull
request with that head commit shares. Pull requests with different bases
have different changed files, so a decision made for one could mark the
other ready without checking it.

Generated-by: Claude Opus 5
Runs from forks carry no pull request, so they were matched by fork branch
alone. Legitimate project CI against a base branch without the gate, made
before a retarget, or a run of another pull request from the same branch,
could close a gated pull request for gate tampering.

Generated-by: Claude Opus 5
A pull request whose CI runs in the contributor's fork could change the gate
that sent it there: its code, its configuration, the gate job or the
monitor. Such a change decides where the pull request's own CI runs, so the
monitor now closes it and explains why. Contributors can still add, change
or remove their own pilot entry, which is how they join and leave the pilot.

Generated-by: Claude Opus 5
@potiuk
potiuk force-pushed the fork-ci-6-reconciler-followups branch from 5ffe91d to 8196f7a Compare October 17, 2026 19:25
@potiuk
potiuk force-pushed the fork-ci-5-docs branch 2 times, most recently from e9d1bb4 to 87df430 Compare October 20, 2026 22:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants