Repository navigation
[flink] Preserve query authorization when narrowing splits in the Flink source - #9949
Merged
JingsongLi merged 1 commit intoSep 20, 2026
Merged
Conversation
plusplusjiajia
force-pushed
the
fix/query-auth-flink-splits
branch
5 times, most recently
from
September 19, 2026 08:05
22b20c2 to
2cbe9fe
Compare
plusplusjiajia
force-pushed
the
fix/query-auth-flink-splits
branch
from
September 19, 2026 08:20
2cbe9fe to
9965ae1
Compare
plusplusjiajia
marked this pull request as ready for review
September 19, 2026 08:30
JingsongLi
reviewed
Sep 20, 2026
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Requirement fit: SUPPORTED (triage: GO)
Implementation: CLEAN
This restores query authorization across the Flink source end to end: code that only inspects split shape looks through QueryAuthSplit and fallback wrappers, while readers and assigners continue carrying the original wrapped split so row filters and masks are not lost. I also checked dynamic partition pruning with masked keys, incremental split routing, alignment, snapshot/watermark metrics, serialization, and chain/fallback reads. The focused tests cover the wrapper combinations and both Flink lines. I did not find an actionable issue, and CI is green.
XiaoHongbo-Hope
pushed a commit
that referenced
this pull request
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
QueryAuthSplitcarries the row filter and the column masks. It implementsSplitbut is not aDataSplit, andAbstractDataTableScan#plan()wraps every split once a table has rules, so the Flink source's casts fail on any table with one. #9929 fixed this class of bug inpaimon-core; this is the same fix for the Flink source.Each site had to answer the same question: does it read the underlying split, or hand the split on to a read? The ones that only read it look through the wrappers now, while the element they route, assign or key on stays the planned split — looking through something passed downstream would drop the rules and return unmasked, unfiltered rows instead of failing.
One wrapper is not enough.
toFallbackSplitproduces aFallbackDataSplit, which is aDataSplitand keeps every cast working — but only when handed aDataSplit. Under authorization it is handed aQueryAuthSplitand produces aFallbackSplitImplinstead, so enabling authorization is what turns a working cast into a failing one.Splits.underlyinglooks through both, stopping at aDataSplit:FallbackDataSplitis one and wraps itself, so that condition ends the loop as much as it preserves what a plan with no rules resolved to.Nine sites, three of which fail quietly rather than loudly:
TableScanUtils#getSnapshotIdandFileStoreSourceSplitReadertest forDataSplitand fall through, reporting no snapshot id and no file creation time — consumer progress and watermarks stop advancing for every split of an authorized table.AlignedSplitAssigner'sinstanceof PlaceholderSplitread the wrapper, so a wrapped placeholder took the wrong branch.ContinuousFileSplitEnumerator,AlignedContinuousFileSplitEnumerator,PreAssignSplitAssigner,DynamicPartitionPruningAssigner,MonitorSource#shuffleOrderedandAlignedSplitAssigner's four alignment reads fail the cast outright.Two needed more than unwrapping.
MonitorSource#shuffleOrderedonly ever handled aDataSplit, so anIncrementalSplitfailed the same cast with or without rules; it keys on partition and bucket for either shape now.ReadOperatorreportsUNDEFINEDfor a split with no file creation time rather than failing a read over a metric it cannot compute, and no longer turns thatUNDEFINEDinto an epoch-sizedcurrentEmitEventTimeLag. That last guard is the one thing here without a test: the value is only observable through the operator's metric group, and intercepting it needs an API whose signature differs between Flink 1 and Flink 2.Making these casts succeed exposed one more way to answer wrong. Dynamic partition pruning runs in the assigners, after planning, and decides from the partition the plan recorded — which a masked partition key makes the wrong value to decide on. A row stored under
p=2but read back asp=1had its split pruned against a dynamic key of{1}and quietly went missing; previously this path crashed on the cast instead. A mask on a filtering field keeps the split now and leaves the join to filter on what the reader produces.QueryAuthSplit.unwrapis removed: a second wrapper meant every caller had to change anyway, soSplits.underlyingreplaces it everywhere, including three sites inChainGroupReadTableandChainTableFileStoreTablethat had the same exposure and had simply not met a fallback-wrapped split yet. Two entry points differing only in how far they look would be the next version of this bug.