Skip to content

[python] Reuse REST table data token for native reader initialization - #10436

Merged
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/reuse-native-rest-token
Oct 8, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/reuse-native-rest-token

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

During multi-worker runs, we observed intermittent DNS resolution failures on REST /token requests, followed by native-plan fallback to the Python reader. The REST-to-native bridge can fetch a second data token for the same table even when PyPaimon already has a valid one, adding avoidable startup requests across workers. A Python token refresh in this bridge is also unsafe after fork if the parent held the global token lock.

Change

Reuse only an already-valid token on the matching Python REST FileIO. The bridge never refreshes a Python token or reads its shared token cache. If no valid instance token exists, Rust obtains its own token; Rust still refreshes after expiry. Older native wheels keep the existing constructor. Requires paimon-rust#1076 for token handoff.

Validation

  • A controlled fork test deadlocked at the previous head and passes after this fix; it holds the parent token lock during child native-cache initialization.
  • Token-present and token-absent paths have unit coverage. Local focused tests: 91 passed. With a Rust wheel containing that token binding (SHA-256 b1f4e3545ae3b00382aef96f060d531090a689498cd5d0a9118701205981be00), the REST native-plan suite passed 20 tests, including token reuse with local cache both off and on. The wheel was built from an equivalent token implementation, not the exact Rust PR commit.
  • Local REST A/B showed 2 token GETs (Python + Rust) versus 1 with reuse. In a same-base 8-rank × 4-worker A/B/B/A, means were 391.0 versus 385.8 samples/s (−1.3%); this does not establish a throughput gain or regression. No natural DNS failures occurred in those rounds.

This reduces redundant cold-start requests. It has not been shown that duplicate requests alone caused the DNS failures, and this change does not share tokens across workers or guarantee DNS recovery.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title Reuse REST table data token for native reader initialization [python] Reuse REST table data token for native reader initialization Oct 8, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 8, 2026 12:44

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit e1d1f60 into apache:master Oct 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants