Skip to content

[python] Fix MAP and MULTISET type JSON serialization - #9682

Merged
JingsongLi merged 2 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/fix-partitioned-self-merge-rebase
Sep 20, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/fix-partitioned-self-merge-rebase

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

MAP and MULTISET fields can cause native planning to fall back to Python:

Invalid table schema JSON: unknown variant `MAP<STRING`
Invalid table schema JSON: unknown variant `MULTISET<INT>`

Fix MapType.to_dict() and MultisetType.to_dict() to emit canonical MAP / MULTISET tags, with NOT NULL when needed. Preserve child types, nullability, and existing legacy JSON parsing.

Validation

  • 312 local tests passed with paimon-rust 4178002a, including MAP native reads and 12 MULTISET Rust schema checks.
  • MULTISET: 5 regression failures and 12 Rust schema failures before the fix; all pass afterward.
  • flake8, license checks, and git diff --check passed.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/fix-partitioned-self-merge-rebase branch from f7bcb57 to 23a47d3 Compare September 18, 2026 14:38
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python][ray] Rebase exact-range self-merge updates [python] Fix MAP type JSON serialization for native schema interoperability Sep 18, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Fix MAP type JSON serialization for native schema interoperability [python] Fix MAP type JSON serialization Sep 18, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 18, 2026 16:04
@JingsongLi

Copy link
Copy Markdown
Contributor

The MAP fix itself looks right: MAP / MAP NOT NULL matches the canonical Java tag (MapType.java:90), the parser accepts both the canonical and legacy MAP<...> forms, and the tests cover the nullability matrix.

One same-class gap is left behind: MultisetType.to_dict() (paimon-python/pypaimon/schema/data_types.py:221-227) still emits the embedded-element form {"type": "MULTISET<...>", "element": ..., "nullable": ...}. Since _resolved_schema_json serializes every field of the table schema, any table with a MULTISET column would hit the same native-plan failure this PR fixes for MAP:

Invalid table schema JSON: unknown variant `MULTISET<...>`,
expected one of `ARRAY`, `MAP`, `MULTISET`, `ROW`, `VECTOR`

The canonical tag already exists on the Java side (MultisetType.java:83 emits MULTISET / MULTISET NOT NULL), and the Python parser's MULTISET branch (data_types.py:573-578) accepts both forms, so switching the tag would be backward compatible. Could you extend the same fix and tests to MULTISET here, or file a follow-up so the native path is not left half-fixed?

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Fix MAP type JSON serialization [python] Fix MAP and MULTISET type JSON serialization Sep 19, 2026
@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor Author

extend the same fix and tests to MULTISET here

Thanks, I extended the same fix and tests to MULTISET here

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit ba091e1 into apache:master Sep 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants