Repository navigation
[python] Reuse REST table data token for native reader initialization - #10436
Merged
JingsongLi merged 3 commits intoOct 8, 2026
Merged
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
October 8, 2026 12:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
During multi-worker runs, we observed intermittent DNS resolution failures on REST
/tokenrequests, followed by native-plan fallback to the Python reader. The REST-to-native bridge can fetch a second data token for the same table even when PyPaimon already has a valid one, adding avoidable startup requests across workers. A Python token refresh in this bridge is also unsafe afterforkif the parent held the global token lock.Change
Reuse only an already-valid token on the matching Python REST FileIO. The bridge never refreshes a Python token or reads its shared token cache. If no valid instance token exists, Rust obtains its own token; Rust still refreshes after expiry. Older native wheels keep the existing constructor. Requires paimon-rust#1076 for token handoff.
Validation
forktest deadlocked at the previous head and passes after this fix; it holds the parent token lock during child native-cache initialization.b1f4e3545ae3b00382aef96f060d531090a689498cd5d0a9118701205981be00), the REST native-plan suite passed 20 tests, including token reuse with local cache both off and on. The wheel was built from an equivalent token implementation, not the exact Rust PR commit.This reduces redundant cold-start requests. It has not been shown that duplicate requests alone caused the DNS failures, and this change does not share tokens across workers or guarantee DNS recovery.