Skip to content

Add consolidated Language SDK compatibility matrix - #71757

Open
jason810496 wants to merge 5 commits into
apache:mainfrom
jason810496:ci/lang-sdk/compat-matrix-common
Open

jason810496 wants to merge 5 commits into
apache:mainfrom
jason810496:ci/lang-sdk/compat-matrix-common

Conversation

@jason810496

@jason810496 jason810496 commented Aug 18, 2026 •

Copy link
Copy Markdown
Member

Why

The Language SDK docs had no single view of what Go, Java and TypeScript each support, so users had to read three READMEs, and the per-SDK manifests drifted from the code.

What

  • One generated matrix in the Language SDK docs, in three parts: features common to both authoring modes, Dag definition, and Python Dag with stub TaskHandler. A supported cell shows the SDK release that shipped it.
  • The authoring modes are dag-definition and stub-task-handler. A capability that exists in both declares one entry per mode and appears in both parts.
  • New Dag-definition dimensions for the primitives the SDKs ship: dag-args, explicit-dependencies, edge-labels, if-else-branching / switch-case-branching (replacing branching), task-loops, trigger-dag-run, dynamic-dag-generation, human-in-the-loop, asset-definition.
  • asset-definition (declaring assets) is kept distinct from asset-scheduling (scheduling on asset events, including the AssetWatcher equivalent) and asset-state-store (the store keyed by an asset, not an authoring surface).
  • New dimensions for gaps every SDK shares today: dag-callbacks (a Dag definition has to declare them itself; a Python Dag's callbacks already run), task-callbacks (declared per mode, since a stub task needs the Python-declared one to run) and listeners (the listener API OpenLineage consumes).
  • A new update-lang-sdk-central-matrix prek hook regenerates the matrix from the manifests, like the per-SDK README hooks.
# <sdk>/capabilities.yaml
edge-labels:
  supported: true
  since: "3.4.0"
  sdk_version: "1.0.0"
mapped-tasks:
  dag-definition:
    supported: false
  stub-task-handler:
    supported: false

Was generative AI tooling used to co-author this PR?
  • Yes, with help from Claude Code (Opus 4.8 and Opus 5) and Codex (GPT-5) following the guidelines

Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated
Users comparing Language SDKs had to read each SDK's README, and the
per-SDK manifests lagged behind what the SDKs actually ship. One generated
matrix, split by authoring mode, shows what each SDK supports for a Dag
definition and for a Python Dag with stub TaskHandler.

Co-Authored-By: Claude <noreply@anthropic.com>
@jason810496
jason810496 force-pushed the ci/lang-sdk/compat-matrix-common branch from a539342 to df39754 Compare October 22, 2026 08:55
Users comparing Language SDKs could not see where callbacks, Airflow's
listener API and Python provider operators stand, because the matrix had
no dimension for any of them. Callbacks ask different things of an SDK per
authoring mode, so they are declared per mode like the other shared
capabilities.

Co-Authored-By: Claude <noreply@anthropic.com>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments are addressed in df39754.

I update the section with task-loops and all the features that I'm able to foresee, please let me know if there's other feature that I didn't mention.

Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread scripts/ci/prek/update_lang_sdk_central_matrix.py
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated
@uranusjr

Copy link
Copy Markdown
Member

PR description is out of date.

Can you set up your Claude skill or whatever to do this automatically going forward? This has happened more than a couple of times at this point and I don’t like checking this every time.

Comment thread java-sdk/capabilities.yaml
Comment thread ts-sdk/capabilities.yaml Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread scripts/tests/ci/prek/test_update_lang_sdk_central_matrix.py Outdated
Comment thread scripts/tests/ci/prek/test_lang_sdk_compat_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
The umbrella capability and the authoring mode both read `dag-definition`,
which gives one name two meanings in the same manifest. A row's kind also
belongs in its entry key: state and capability names happen not to collide
today, so the lookup worked by accident rather than by design.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst Outdated
The dimension is about whether a task can be a mapped task, which the
shorter name says directly and keeps the matrix row readable next to the
other task-level dimensions.

Co-Authored-By: Claude <noreply@anthropic.com>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments are addressed in 4b7c04f, thanks for the review.

PR description is out of date.
Can you set up your Claude skill or whatever to do this automatically going forward? This has happened more than a couple of times at this point and I don’t like checking this every time.

Actually, I asked CC to do it explicitly right before I claim the PR is ready for review.

Comment thread java-sdk/capabilities.yaml
Comment thread ts-sdk/capabilities.yaml Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread scripts/tests/ci/prek/test_update_lang_sdk_central_matrix.py Outdated
Comment thread scripts/tests/ci/prek/test_lang_sdk_compat_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst Outdated
Comment thread scripts/tests/ci/prek/test_update_go_sdk_readme_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated
The callbacks of a Python Dag run whichever SDK executes its stub tasks, so
``dag-callbacks`` is a Dag-definition concern only, while ``task-callbacks``
stays declared per authoring mode. Drop ``provider-operators`` until ADR-0009
is accepted, and reuse the shared ``capabilities_doc`` fixture in the per-SDK
README tests.

Co-Authored-By: Claude <noreply@anthropic.com>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread scripts/tests/ci/prek/test_update_go_sdk_readme_matrix.py Outdated
Comment thread scripts/ci/prek/lang_sdk_compat_matrix.py Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated
Comment thread contributing-docs/30_new_language_sdk.rst Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants