Skip to content

Fix task loop information and tests for OpenLineage provider - #74752

Open
kacpermuda wants to merge 27 commits into
apache:mainfrom
kacpermuda:ol-loop-iteration-facet-and-run-id-check
Open

kacpermuda wants to merge 27 commits into
apache:mainfrom
kacpermuda:ol-loop-iteration-facet-and-run-id-check

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Task loops (Airflow 3.4) and mapped tasks turn one task into many executions sharing the Dag, task and run ids. Building on #75132, each OpenLineage task event now tells which execution sent it, without querying Airflow:

  • taskInstance: id, state, max_tries, dag_version_id and bundle, expanded_ti_count, region_id/region_index, and loop (iteration, rerun trajectory, gate decision).
  • task: loop (the loop around the task, also from a task group inside it), task_group.unique_group_id, retry_delay.
  • Job facet: per task loop_group, is_mapped, parse_time_mapped_count, trigger_rule, depends_on_past, retries; per group is_mapped, is_loop and the loop definition.

A FAIL the scheduler sends itself, e.g. after a worker died, is built without extra queries: from the task the scheduler holds, or from the task instance's region rows when that task comes from another Dag version. It shares the runId of the worker's START and carries the attempt's id (3.4+), state, map index, bundle and, for a task that isn't mapped, loop.index. expanded_ti_count and the loop run fields (loop.region_id, trajectory, fork fields) stay in worker events, as AirflowRunFacet.json describes. UI/API state changes still have no airflow facet.

Other changes consumers can notice:

  • The scheduler's FAIL for a mapped task instance is now sent; reading the mapped operator's documentation fields used to fail it.
  • That FAIL carries rendered_map_index only when the task renders one with map_index_template, as the worker's events do, instead of falling back to the map index.
  • taskInstance.note is no longer sent: it only reached scheduler-sent FAILs, at the cost of a query on the scheduler's session.

The log link of a task instance in a loop is fixed in Airflow itself.

related: #75458

System tests check runId integrity by default, including invariants of loop events, and cover retries, rescheduled sensors, task states, and every task group / loop arrangement, also in the e2e suite.


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code (Opus 5.5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

🤖 Generated with Claude Code

@kacpermuda
kacpermuda force-pushed the ol-loop-iteration-facet-and-run-id-check branch from fb42765 to 69b1b1e Compare October 16, 2026 14:33
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #74290 ("Add AzureAISearchHook to the Microsoft Azure provider"), commit 935bac0 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@kacpermuda
kacpermuda force-pushed the ol-loop-iteration-facet-and-run-id-check branch from 6e892fd to 54821f3 Compare October 19, 2026 11:40
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
Comment thread providers/openlineage/tests/system/openlineage/operator.py
Comment thread providers/openlineage/src/airflow/providers/openlineage/utils/utils.py Outdated
@kacpermuda
kacpermuda force-pushed the ol-loop-iteration-facet-and-run-id-check branch 3 times, most recently from 76490cd to 72264e0 Compare October 21, 2026 13:13
@kacpermuda kacpermuda changed the title Improve task loop information and tests for OpenLineage provider Fix task loop information and tests for OpenLineage provider Oct 21, 2026
@kacpermuda

Copy link
Copy Markdown
Collaborator Author

Finishing up last test fixes and additions, should be ready for re-review soon

@kacpermuda
kacpermuda force-pushed the ol-loop-iteration-facet-and-run-id-check branch from b0defb5 to 196dae7 Compare October 23, 2026 13:36
kacpermuda and others added 27 commits October 23, 2026 16:47
Provider mypy and the docs spellcheck run in CI through breeze rather than
the default prek hooks, so they were not caught locally before the previous push.

Co-Authored-By: Claude <noreply@anthropic.com>
Alerting on why a loop stopped (for example, its until condition still being false at
max_iterations) needs the gate's decision in the event itself, without reading the Airflow
metadata database.

Co-Authored-By: Claude <noreply@anthropic.com>
The run id must not depend on the optional loop iteration, and the lineage macro should
place a task instance the same way the listener does.

Co-Authored-By: Claude <noreply@anthropic.com>
A region of an unexpected shape, or a failed lookup of a mapped task's iteration, must not
merge the runs of loop iterations or cost an event. Tests now pin the region shapes Airflow
creates, so a change to them fails the provider's tests instead of silently changing events.

Co-Authored-By: Claude <noreply@anthropic.com>
The dbt Cloud and Dataproc tests patched how OpenLineage builds a task's run id, so they broke
when that changed. Stubbing the macros they actually use keeps them independent of it.

Co-Authored-By: Claude <noreply@anthropic.com>
Uses what apache#75132 exposes, so a single event tells which rerun of a
loop it comes from, whether that rerun superseded an earlier one, and how many
mapped instances to expect, and events for manual state changes get the same
details as worker events. Loop details are grouped in one object so it can
describe enclosing loops if Airflow ever nests them, and iterations are counted
from one as in Airflow.

Co-Authored-By: Claude <noreply@anthropic.com>
Stored task instances have no region_id before Airflow 3.4, so reading it before
checking the version failed the event.

Co-Authored-By: Claude <noreply@anthropic.com>
…code

The API server holds task instances without their task when a user sets a
state, so those events never carried the airflow run facet the guide promised.
The fallbacks for loop details only served Airflow builds from before
apache#75132, which no release contains. Timing the stored position
lookup makes its per task instance queries in the scheduler visible.

Co-Authored-By: Claude <noreply@anthropic.com>
Placing a task instance pinned to another Dag version than the one the
scheduler holds made Airflow load and deserialize that version again for
every such task instance, inside the scheduler loop. Those now fall back to
reading their region rows. That bounds the cost instead of only measuring
it, so the timer added for it goes away before any release has it.

Co-Authored-By: Claude <noreply@anthropic.com>
The guide will be written separately, so this change only keeps the facet
schema as the description of the new fields.

Co-Authored-By: Claude <noreply@anthropic.com>
…ents

Consumers placing a task event need its state to tell a retry from a final
failure, and the Dag version it ran, from the event alone. They also need the
same fields whether the worker or the scheduler sent it, and each fact in one
place: a loop's definition that also showed in the task's own group read as
two things that could disagree.

Co-Authored-By: Claude <noreply@anthropic.com>
…acet

Consumers that prepare for a Dag run's task events, for example to alert on a
task group, need to know before the tasks run which of them expand, which
groups are mapped or loops, and what lets a task run or keeps it from running.

Co-Authored-By: Claude <noreply@anthropic.com>
…Lineage tests

The provider's compatibility tests against Airflow 3.0 failed, as its runtime
task instances don't carry a Dag version, which the event then leaves out.

Co-Authored-By: Claude <noreply@anthropic.com>
Task events place a task by its own group, the loop around it and its
iteration, which differ by where the task sits: in a plain or nested group, in
a mapped group, in a group inside a loop, or in a loop that is itself in a
group. Each arrangement now has a task whose events, and the Dag's job facet,
the e2e run checks.

Co-Authored-By: Claude <noreply@anthropic.com>
Reading a stored task instance's Dag version can query the database, and a
failure there dropped the whole event the scheduler sent about it. The bundle
name and version only describe the event, so it now goes out without them.

Co-Authored-By: Claude <noreply@anthropic.com>
…stem tests

The facet schemas only describe fields loosely, as they guarantee none, so
they now add plain types. Loop groups tell independently whether they're
mapped, as a loop may become mappable, and fields read from Airflow objects
tolerate versions that lack them. System tests check every field of the new
events, per Airflow version where it differs, instead of skipping optional
ones, and Airflow 2 expects a literal expansion's size.

Co-Authored-By: Claude <noreply@anthropic.com>
The id Airflow identifies a group by is its own id prefixed by its parents'
only where they prefix their children, so it isn't always a full path.

Co-Authored-By: Claude <noreply@anthropic.com>
…low 3.2

The provider's compatibility tests against Airflow 3.0 and 3.1 failed, as their
Task SDK operators still count a literal expansion before serialization.

Co-Authored-By: Claude <noreply@anthropic.com>
…2e tests

Airflow 3.0 calls no listener when a task skips itself, so the provider can't
emit the event there, which failed the e2e compatibility test.

Co-Authored-By: Claude <noreply@anthropic.com>
…e2e tests

Without a COMPLETE, the START alone fails the runId check on Airflow 3.0, where
no listener runs for a task that skips itself.

Co-Authored-By: Claude <noreply@anthropic.com>
Reading the loop context, Dag version and expansion size inside the
scheduler's failure handling flushed its pending changes through savepoints
and cost several queries per failed task instance. A Dag edited after an
attempt started could also give its FAIL another runId than its START, the
stored log_url pointed a loop iteration at a mapped index, and before
Airflow 3.4 the attempt already held its next id when the event was built.
The system tests could not notice a disabled runId check, loop fields that
contradict each other, or a checking task that retries and hides the first
mismatch.

Co-Authored-By: Claude <noreply@anthropic.com>
The note was meant to show manual state changes, but on Airflow 3 their
events carry no airflow facet, so it never reached them. Only the FAIL the
scheduler sends itself read it, at the cost of a query on the scheduler's
session for every task instance it fails, and Airflow 2 workers read it with
a query for every task event.

Co-Authored-By: Claude <noreply@anthropic.com>
Airflow 3.4.0, the first release with task loops, links a loop's task
instances to their own page (apache#75458), so rewriting the link in the provider
would only have patched the 3.4.0 pre-releases.

Co-Authored-By: Claude <noreply@anthropic.com>
Consumers could not tell from the schema why an event carries both region
fields, though together they are what Airflow's REST API and UI select a
task instance in a loop or expansion by.

Co-Authored-By: Claude <noreply@anthropic.com>
On the scheduler a mapped task is a serialized mapped operator, which leaves
the documentation fields its task doesn't set undefined. Reading them failed
the event, so a mapped task instance the scheduler failed, e.g. after its
worker died, never got a FAIL to close its run.

Co-Authored-By: Claude <noreply@anthropic.com>
The loop Dags defined the same event order twice, and five Dag docstrings
repeated what a task event's identity covers. A test also promised to check
the loop's max_iterations, which it never did.

Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda force-pushed the ol-loop-iteration-facet-and-run-id-check branch from abb1dc1 to 1d3820f Compare October 23, 2026 14:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants