Repository navigation
Fix task loop information and tests for OpenLineage provider - #74752
Open
kacpermuda wants to merge 27 commits into
Open
kacpermuda wants to merge 27 commits into
kacpermuda wants to merge 27 commits into
Conversation
kacpermuda
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl,
mobuchowski and
potiuk
as code owners
October 14, 2026 16:52
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
from
October 14, 2026 17:45
ad8bf00 to
fb42765
Compare
ashb
reviewed
Oct 14, 2026
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
from
October 16, 2026 14:33
fb42765 to
69b1b1e
Compare
Contributor
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
from
October 19, 2026 11:40
6e892fd to
54821f3
Compare
mobuchowski
reviewed
Oct 19, 2026
This was referenced Oct 19, 2026
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
3 times, most recently
from
October 20, 2026 13:19
c2d487f to
18e5856
Compare
ashb
reviewed
Oct 20, 2026
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
3 times, most recently
from
October 21, 2026 13:13
76490cd to
72264e0
Compare
Collaborator
Author
|
Finishing up last test fixes and additions, should be ready for re-review soon |
10 of 93 tasks
kacpermuda
marked this pull request as draft
October 21, 2026 18:08
1 task done
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
from
October 23, 2026 13:36
b0defb5 to
196dae7
Compare
Provider mypy and the docs spellcheck run in CI through breeze rather than the default prek hooks, so they were not caught locally before the previous push. Co-Authored-By: Claude <noreply@anthropic.com>
Alerting on why a loop stopped (for example, its until condition still being false at max_iterations) needs the gate's decision in the event itself, without reading the Airflow metadata database. Co-Authored-By: Claude <noreply@anthropic.com>
The run id must not depend on the optional loop iteration, and the lineage macro should place a task instance the same way the listener does. Co-Authored-By: Claude <noreply@anthropic.com>
A region of an unexpected shape, or a failed lookup of a mapped task's iteration, must not merge the runs of loop iterations or cost an event. Tests now pin the region shapes Airflow creates, so a change to them fails the provider's tests instead of silently changing events. Co-Authored-By: Claude <noreply@anthropic.com>
The dbt Cloud and Dataproc tests patched how OpenLineage builds a task's run id, so they broke when that changed. Stubbing the macros they actually use keeps them independent of it. Co-Authored-By: Claude <noreply@anthropic.com>
Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>
Stored task instances have no region_id before Airflow 3.4, so reading it before checking the version failed the event. Co-Authored-By: Claude <noreply@anthropic.com>
…code The API server holds task instances without their task when a user sets a state, so those events never carried the airflow run facet the guide promised. The fallbacks for loop details only served Airflow builds from before apache#75132, which no release contains. Timing the stored position lookup makes its per task instance queries in the scheduler visible. Co-Authored-By: Claude <noreply@anthropic.com>
Placing a task instance pinned to another Dag version than the one the scheduler holds made Airflow load and deserialize that version again for every such task instance, inside the scheduler loop. Those now fall back to reading their region rows. That bounds the cost instead of only measuring it, so the timer added for it goes away before any release has it. Co-Authored-By: Claude <noreply@anthropic.com>
The guide will be written separately, so this change only keeps the facet schema as the description of the new fields. Co-Authored-By: Claude <noreply@anthropic.com>
…ents Consumers placing a task event need its state to tell a retry from a final failure, and the Dag version it ran, from the event alone. They also need the same fields whether the worker or the scheduler sent it, and each fact in one place: a loop's definition that also showed in the task's own group read as two things that could disagree. Co-Authored-By: Claude <noreply@anthropic.com>
…acet Consumers that prepare for a Dag run's task events, for example to alert on a task group, need to know before the tasks run which of them expand, which groups are mapped or loops, and what lets a task run or keeps it from running. Co-Authored-By: Claude <noreply@anthropic.com>
…Lineage tests The provider's compatibility tests against Airflow 3.0 failed, as its runtime task instances don't carry a Dag version, which the event then leaves out. Co-Authored-By: Claude <noreply@anthropic.com>
Task events place a task by its own group, the loop around it and its iteration, which differ by where the task sits: in a plain or nested group, in a mapped group, in a group inside a loop, or in a loop that is itself in a group. Each arrangement now has a task whose events, and the Dag's job facet, the e2e run checks. Co-Authored-By: Claude <noreply@anthropic.com>
Reading a stored task instance's Dag version can query the database, and a failure there dropped the whole event the scheduler sent about it. The bundle name and version only describe the event, so it now goes out without them. Co-Authored-By: Claude <noreply@anthropic.com>
…stem tests The facet schemas only describe fields loosely, as they guarantee none, so they now add plain types. Loop groups tell independently whether they're mapped, as a loop may become mappable, and fields read from Airflow objects tolerate versions that lack them. System tests check every field of the new events, per Airflow version where it differs, instead of skipping optional ones, and Airflow 2 expects a literal expansion's size. Co-Authored-By: Claude <noreply@anthropic.com>
The id Airflow identifies a group by is its own id prefixed by its parents' only where they prefix their children, so it isn't always a full path. Co-Authored-By: Claude <noreply@anthropic.com>
…low 3.2 The provider's compatibility tests against Airflow 3.0 and 3.1 failed, as their Task SDK operators still count a literal expansion before serialization. Co-Authored-By: Claude <noreply@anthropic.com>
…2e tests Airflow 3.0 calls no listener when a task skips itself, so the provider can't emit the event there, which failed the e2e compatibility test. Co-Authored-By: Claude <noreply@anthropic.com>
…e2e tests Without a COMPLETE, the START alone fails the runId check on Airflow 3.0, where no listener runs for a task that skips itself. Co-Authored-By: Claude <noreply@anthropic.com>
Reading the loop context, Dag version and expansion size inside the scheduler's failure handling flushed its pending changes through savepoints and cost several queries per failed task instance. A Dag edited after an attempt started could also give its FAIL another runId than its START, the stored log_url pointed a loop iteration at a mapped index, and before Airflow 3.4 the attempt already held its next id when the event was built. The system tests could not notice a disabled runId check, loop fields that contradict each other, or a checking task that retries and hides the first mismatch. Co-Authored-By: Claude <noreply@anthropic.com>
The note was meant to show manual state changes, but on Airflow 3 their events carry no airflow facet, so it never reached them. Only the FAIL the scheduler sends itself read it, at the cost of a query on the scheduler's session for every task instance it fails, and Airflow 2 workers read it with a query for every task event. Co-Authored-By: Claude <noreply@anthropic.com>
Airflow 3.4.0, the first release with task loops, links a loop's task instances to their own page (apache#75458), so rewriting the link in the provider would only have patched the 3.4.0 pre-releases. Co-Authored-By: Claude <noreply@anthropic.com>
Consumers could not tell from the schema why an event carries both region fields, though together they are what Airflow's REST API and UI select a task instance in a loop or expansion by. Co-Authored-By: Claude <noreply@anthropic.com>
On the scheduler a mapped task is a serialized mapped operator, which leaves the documentation fields its task doesn't set undefined. Reading them failed the event, so a mapped task instance the scheduler failed, e.g. after its worker died, never got a FAIL to close its run. Co-Authored-By: Claude <noreply@anthropic.com>
The loop Dags defined the same event order twice, and five Dag docstrings repeated what a task event's identity covers. A test also promised to check the loop's max_iterations, which it never did. Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda
force-pushed
the
ol-loop-iteration-facet-and-run-id-check
branch
from
October 23, 2026 14:49
abb1dc1 to
1d3820f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task loops (Airflow 3.4) and mapped tasks turn one task into many executions sharing the Dag, task and run ids. Building on #75132, each OpenLineage task event now tells which execution sent it, without querying Airflow:
taskInstance:id,state,max_tries,dag_version_idand bundle,expanded_ti_count,region_id/region_index, andloop(iteration, rerun trajectory, gate decision).task:loop(the loop around the task, also from a task group inside it),task_group.unique_group_id,retry_delay.loop_group,is_mapped,parse_time_mapped_count,trigger_rule,depends_on_past,retries; per groupis_mapped,is_loopand the loop definition.A FAIL the scheduler sends itself, e.g. after a worker died, is built without extra queries: from the task the scheduler holds, or from the task instance's region rows when that task comes from another Dag version. It shares the runId of the worker's START and carries the attempt's
id(3.4+),state, map index, bundle and, for a task that isn't mapped,loop.index.expanded_ti_countand the loop run fields (loop.region_id,trajectory, fork fields) stay in worker events, asAirflowRunFacet.jsondescribes. UI/API state changes still have noairflowfacet.Other changes consumers can notice:
rendered_map_indexonly when the task renders one withmap_index_template, as the worker's events do, instead of falling back to the map index.taskInstance.noteis no longer sent: it only reached scheduler-sent FAILs, at the cost of a query on the scheduler's session.The log link of a task instance in a loop is fixed in Airflow itself.
related: #75458
System tests check
runIdintegrity by default, including invariants of loop events, and cover retries, rescheduled sensors, task states, and every task group / loop arrangement, also in the e2e suite.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.🤖 Generated with Claude Code