Repository navigation
Add consolidated Language SDK compatibility matrix - #71757
jason810496 wants to merge 5 commits into
Conversation
2bc4028 to
8c476ab
Compare
81c5249 to
031d2bc
Compare
cdd3c79 to
a539342
Compare
Users comparing Language SDKs had to read each SDK's README, and the per-SDK manifests lagged behind what the SDKs actually ship. One generated matrix, split by authoring mode, shows what each SDK supports for a Dag definition and for a Python Dag with stub TaskHandler. Co-Authored-By: Claude <noreply@anthropic.com>
a539342 to
df39754
Compare
Users comparing Language SDKs could not see where callbacks, Airflow's listener API and Python provider operators stand, because the matrix had no dimension for any of them. Callbacks ask different things of an SDK per authoring mode, so they are declared per mode like the other shared capabilities. Co-Authored-By: Claude <noreply@anthropic.com>
jason810496
left a comment
There was a problem hiding this comment.
All the comments are addressed in df39754.
I update the section with task-loops and all the features that I'm able to foresee, please let me know if there's other feature that I didn't mention.
|
PR description is out of date. Can you set up your Claude skill or whatever to do this automatically going forward? This has happened more than a couple of times at this point and I don’t like checking this every time. |
The umbrella capability and the authoring mode both read `dag-definition`, which gives one name two meanings in the same manifest. A row's kind also belongs in its entry key: state and capability names happen not to collide today, so the lookup worked by accident rather than by design. Co-Authored-By: Claude <noreply@anthropic.com>
The dimension is about whether a task can be a mapped task, which the shorter name says directly and keeps the matrix row readable next to the other task-level dimensions. Co-Authored-By: Claude <noreply@anthropic.com>
jason810496
left a comment
There was a problem hiding this comment.
All the comments are addressed in 4b7c04f, thanks for the review.
PR description is out of date.
Can you set up your Claude skill or whatever to do this automatically going forward? This has happened more than a couple of times at this point and I don’t like checking this every time.
Actually, I asked CC to do it explicitly right before I claim the PR is ready for review.
The callbacks of a Python Dag run whichever SDK executes its stub tasks, so ``dag-callbacks`` is a Dag-definition concern only, while ``task-callbacks`` stays declared per authoring mode. Drop ``provider-operators`` until ADR-0009 is accepted, and reuse the shared ``capabilities_doc`` fixture in the per-SDK README tests. Co-Authored-By: Claude <noreply@anthropic.com>
jason810496
left a comment
There was a problem hiding this comment.
Thanks for the review.
Why
The Language SDK docs had no single view of what Go, Java and TypeScript each support, so users had to read three READMEs, and the per-SDK manifests drifted from the code.
What
dag-definitionandstub-task-handler. A capability that exists in both declares one entry per mode and appears in both parts.dag-args,explicit-dependencies,edge-labels,if-else-branching/switch-case-branching(replacingbranching),task-loops,trigger-dag-run,dynamic-dag-generation,human-in-the-loop,asset-definition.asset-definition(declaring assets) is kept distinct fromasset-scheduling(scheduling on asset events, including theAssetWatcherequivalent) andasset-state-store(the store keyed by an asset, not an authoring surface).dag-callbacks(a Dag definition has to declare them itself; a Python Dag's callbacks already run),task-callbacks(declared per mode, since a stub task needs the Python-declared one to run) andlisteners(the listener API OpenLineage consumes).update-lang-sdk-central-matrixprek hook regenerates the matrix from the manifests, like the per-SDK README hooks.Was generative AI tooling used to co-author this PR?