Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions contributing-docs/03_contributors_quick_start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,19 @@ Forking and cloning Project
to clone the repo locally (you can also do it in your IDE - see the `Using your IDE`_
chapter below)

4. Add ``apache/airflow`` as the ``upstream`` remote, following the
`Git remote naming conventions <10_working_with_git.rst>`__:

.. code-block:: bash
git remote add upstream https://github.com/apache/airflow.git
Comment thread
potiuk marked this conversation as resolved.
If you use SSH with GitHub, add ``git@github.com:apache/airflow.git`` instead; ``breeze ci audit``
recognises both forms.

Once Breeze is installed, run ``breeze ci audit`` before your first push to check your remotes
and the CI in your fork; see `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__.

.. note::
For windows based machines, on cloning, the Git line endings may be different from unix based systems
and might lead to unexpected behaviour on running breeze tooling. Manually setting a property will mitigate this issue.
Expand Down
5 changes: 5 additions & 0 deletions contributing-docs/03a_contributors_quick_start_beginners.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ Option A – Breeze on Your Laptop

git clone https://github.com/<you>/airflow.git
cd airflow
git remote add upstream https://github.com/apache/airflow.git

Before your first push, run ``breeze ci audit`` (after installing Breeze below) to check that your
remotes and the CI in your fork are set up; see
`CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__.

2. **Set up Git hooks and install Breeze**

Expand Down
10 changes: 8 additions & 2 deletions contributing-docs/05_pull_requests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,14 @@ these guidelines:

The Airflow repo uses `GitHub Actions <https://help.github.com/en/actions>`_ to
run the tests and `codecov <https://codecov.io/gh/apache/airflow>`_ to track
coverage. You can set up both for free on your fork. It will help you make sure you do not
break the build with your PR and that you help increase coverage.
coverage. Fork CI is being piloted: it applies to pilot participants and to pull requests a
maintainer labels ``use fork ci``, and to every external contributor only once it is switched
on for everyone. Until then, enabling the ``Tests (AMD)`` workflow in your fork outside the
pilot runs CI twice for each pull request, in your fork and in apache/airflow. When fork CI
applies to you, enable that workflow and every push to your fork runs CI for free; pull
requests labelled ``fork ci required`` take their CI result from your fork and stay in draft
until that run is green. See `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__
and run ``breeze ci audit``, which tells you whether fork CI applies to you yet.
Also we advise to install locally `prek hooks <08_static_code_checks.rst#prek-hooks>`_ to
apply various checks, code generation and formatting at the time you make a local commit - which
gives you near-immediate feedback on things you need to fix before you push your code to the PR, or in
Expand Down
3 changes: 3 additions & 0 deletions contributing-docs/10_working_with_git.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ you should configure your local checkout to match:
* ``origin`` — your personal fork of ``apache/airflow`` (where you push branches
for PRs).

``breeze ci audit`` checks these names, offers the commands below when they differ, and also checks
the CI in your fork; see `CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__.

Always push PR branches to ``origin``; don't push to ``upstream`` (the branch
protection on ``apache/airflow`` will reject it anyway). Working on dedicated
branches is recommended, though developing directly on your fork's ``main`` is
Expand Down
65 changes: 64 additions & 1 deletion contributing-docs/18_contribution_workflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ In general, your contribution includes the following stages:
2. Create a `local virtualenv <07_local_virtualenv.rst>`_,
initialize the `Breeze environment <../dev/breeze/doc/README.rst>`__, and
install `prek tool <08_static_code_checks.rst#prek-hooks>`__.
If you want to add more changes in the future, set up your fork and enable GitHub Actions.
Set up the ``upstream`` and ``origin`` remotes and check them with ``breeze ci audit``; if
fork CI applies to you (see `CI in your fork`_), enable CI in your fork as described there.

3. Join `devlist <https://lists.apache.org/list.html?dev@airflow.apache.org>`__
and set up a `Slack account <https://s.apache.org/airflow-slack>`__.
Expand Down Expand Up @@ -126,6 +127,68 @@ to make them immediately visible in the environment.
6. Open your IDE (for example, PyCharm) and select the virtual environment you created
as the project's default virtual environment in your IDE.

CI in your fork
~~~~~~~~~~~~~~~

Pull requests from contributors who are not committers or collaborators are moving to running
their CI in the contributor's own fork, on GitHub's free runners, instead of on the shared CI
capacity of the Apache Software Foundation (`AIP-120
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks>`__).
CI on a public fork costs you nothing and has no minute limit; it runs on every push to your
fork, so you get feedback before a maintainer looks at your pull request. The change is being
piloted: until it is switched on for everyone, it applies to pilot participants (see below) and
to pull requests a maintainer labels ``use fork ci``.

Set up your checkout and fork once:

1. Name your remotes as described in `Git remote naming conventions <10_working_with_git.rst>`__:
``upstream`` for ``apache/airflow`` and ``origin`` for your fork.

2. If fork CI applies to you, enable the ``Tests (AMD)`` workflow in your fork: today that means
you joined the pilot or a maintainer labelled your pull request ``use fork ci``; once fork CI is
switched on for everyone, it means you are not a committer, a collaborator or a member of an
exempted organisation. GitHub disables workflows in new forks. Otherwise keep it disabled, so
your pushes do not run CI twice:

.. code-block:: bash
gh workflow enable ci-amd.yml -R <you>/airflow
If the workflow is missing, your fork's ``main`` is older than it; sync it first with
``gh repo sync <you>/airflow --branch main``.

3. Check everything at once. ``breeze ci audit`` prints one row per check and the command that fixes
anything missing; ``--fix`` runs those commands after asking:

.. code-block:: bash
breeze ci audit
When fork CI applies to your pull request, it is labelled ``fork ci required`` and kept in draft
until the ``Tests (AMD)`` run in your fork for its latest commit is green. A
``Fork CI / Tests (AMD)`` status on the pull request links to that run, and one comment on the pull
request explains the current state. Once the run is green, a pull request the monitor drafted is
marked ready for review and labelled ``ready for maintainer review``; a pull request you drafted
yourself stays a draft, and gets the label, once you mark it ready. A pull request whose fork CI
does not run for 7 days is closed with an explanation; you can reopen it after setting up fork CI. If your pull request changes
files that define what CI runs, such as ``.github/``, ``dev/breeze/`` or ``scripts/ci/``, a green
fork run is not enough: a maintainer reviews the change and runs project CI for it.

**Joining the pilot.** Open a pull request that adds your GitHub login to the ``[pilot]`` table in
``.github/fork-ci-gate.toml``, with a join instant after the expected merge, for example the next day
at 00:00 UTC:

.. code-block:: toml
[pilot]
"your-login" = YYYY-MM-DDT00:00:00Z
Every pull request you open after that instant uses fork CI; your earlier pull requests keep project
CI. To leave the pilot, open a pull request that removes the line. A pilot entry gates your pull
requests whatever your role, so a committer in the pilot also enables the workflow;
``breeze ci audit`` tells you which applies to you. The design is described in
`Fork CI for external contributors <../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md>`__.

Step 3: Connect with People
---------------------------

Expand Down
33 changes: 33 additions & 0 deletions contributing-docs/25_maintainer_pr_triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,39 @@ The triage workflow uses the following labels and states:
| `suspicious changes detected` | PR (and all open PRs by the same author) was closed because the diff contained suspicious patterns (secret exfiltration, malicious CI modifications, etc.). |
| `closed because of open PR limit` | PR was closed in the one-time closure that introduced the [open PR limit](32_open_pull_request_limit.rst). Applied by `dev/close_prs_over_open_pr_limit.py`, never by the triage skill. If the author reopens the PR, triage it like any other. |
| **Draft status** | PR was converted to draft because it does not meet quality criteria. The contributor is asked to fix the listed issues and mark the PR as "Ready for review" once done. |
| `fork ci required` | PR runs its CI in the contributor's fork (AIP-120). Applied and managed by the fork CI reconciler, never by the triage skill. |
| `needs project ci` | Fork CI is green, but the PR changes CI-defining files, so a maintainer must review them and add `use project ci`. Applied and removed by the fork CI reconciler. Filter on it to find PRs waiting only on a maintainer. |
| `use project ci` / `use fork ci` | Set by a maintainer to override the fork CI decision for one PR: run project CI, or use fork CI for a PR that would otherwise run project CI. |

## Pull requests gated to fork CI

Pull requests labelled `fork ci required` run their CI in the contributor's fork. The scheduled
fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns four things
for them, and the triage skill must treat those pull requests as out of scope for each:

- **Draft state.** The reconciler keeps the pull request in draft until the fork run for its
latest commit is green, and undrafts only what it drafted.
- **`ready for maintainer review`.** The reconciler adds it when the fork run is green and the
pull request is not a draft the author made (it adds it once the author marks it ready), and
removes it when a new push starts a new run.
- **`needs project ci`.** The reconciler adds it when the fork run is green but the pull request
changes CI-defining files, and removes it once the state is anything else. These pull requests
stay in draft and wait for a maintainer to review the CI change and add `use project ci`.
- **Closing after inactivity.** The reconciler closes a pull request whose fork CI has not run for
`close_after_days`.

Everything else the skill does (quality criteria, pings, suspicious-change flags) applies as
usual. The `Fork CI / Tests (AMD)` commit status shows the fork CI state and links to the run in
the fork. To move a single pull request back to project CI, add `use project ci`; the reconciler
releases it on its next run; removing `fork ci required` by hand is not enough, the next run
labels a gated pull request again. Adding the label does not start project CI by itself: re-run
the pull request's latest `Tests (AMD)` run (`gh run rerun <run-id> -R apache/airflow`, or
**Re-run all jobs** on the run's page), or ask the author to push. Pull requests that change CI-defining files stay in draft even
when the fork run is green and are labelled `needs project ci`, for a maintainer to review and run
project CI with `use project ci`. When the fork run is green and trusted, the reconciler also posts
the `CI passed` status the merge ruleset requires, so these pull requests can be merged or
auto-merged without project CI. See
[Fork CI for external contributors](../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md).

## For contributors

Expand Down
25 changes: 25 additions & 0 deletions dev/breeze/doc/ci/05_workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@

- [CI run types](#ci-run-types)
- [Pull request run](#pull-request-run)
- [Fork push run](#fork-push-run)
- [Canary run](#canary-run)
- [Workflows](#workflows)
- [Workflow Architecture Overview](#workflow-architecture-overview)
Expand Down Expand Up @@ -92,6 +93,30 @@ all dependencies to latest versions and build images with them. This way we chec
dependencies are upgraded. This can also be forced by setting the `upgrade to newer dependencies`
label in the PR if you are a committer and want to force dependency upgrade.

Every pull request run of `ci-amd.yml` starts with the `Fork CI gate` job. For pull requests
whose CI runs in the contributor's fork (AIP-120), the gate skips `build-info` and therefore
everything after it, so the run finishes in seconds; see
[Fork CI for external contributors](08_fork_ci_for_external_contributors.md).

## Fork push run

These runs are triggered by a push to any branch of a contributor's fork in which the
`Tests (AMD)` workflow is enabled. They run in the fork, on GitHub's public runners and with the
fork's token, and are how pull requests labelled `fork ci required` get their CI result. They are
selected like a pull request run: the changed files come from comparing the pushed commit with
`apache/airflow` (`main`, or the release branch the pushed code was cut from, as breeze's default
branch at that commit records it), so a fork whose `main` is behind does not make the run bigger,
and they never use self-hosted runners. The `schedule` trigger is skipped in forks, so enabling
the workflow does not start the canary matrix there. The fork CI reconciler reads the result of these runs and reports it on the pull
request as the `Fork CI / Tests (AMD)` status.

In `apache/airflow` itself, pushes still only run CI for the release-prep and providers branches
(`CANONICAL_PUSH_BRANCH_FILTERS` in `fork_ci_gate.py`, the branches `release-branch-ci.yml` starts
full runs for), and they test only the changes since the last successful run of the branch. That
incremental base is looked up only in `apache/airflow`: a fork's push is always compared with
`apache/airflow` as described above. A push to any other branch produces a run in which only the
gate executes.

## Canary run

This workflow runs on a schedule on the `main` branch, and a few times a day
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -818,7 +818,6 @@ as a failed check rather than a crash.
- `contributing-docs/10_working_with_git.rst`: cross-link from the naming conventions.
- `contributing-docs/25_maintainer_pr_triage.md`: the ownership split with the reconciler.
- `dev/breeze/doc/ci/05_workflows.md`: add the fork push run type.
- `dev/breeze/doc/ci/03_github_variables.md`: the new gate job outputs.
- `dev/breeze/doc/08_ci_tasks.rst`: `breeze ci audit` and `breeze ci fork-ci-reconcile`.
- `dev/breeze/doc/ci/README.md`: index this document.

Expand Down