[Python] Preserve parallelism options for native global index builds

[JingsongLi]

Dependency

Depends on apache/paimon-rust#1094. Keep this PR in draft until that change is merged. Native CI continues building Apache paimon-rust from main.

What changes

The Python local full-text/vector builder already supports global-index.build.parallelism. The accompanying Rust change enables that concurrency in native builds, retaining independent shard writers, logical ranges, relative row IDs and private preparation ownership.

• Resolve effective parallelism through the existing Python integer ConfigOption conversion before passing it to Rust. This preserves typed inputs, build-over-table precedence and explicit default overrides without Python row processing.
• Include the option name in conversion errors, before any index output is created.
• Document native shard concurrency and commit-message file ownership.
• Add REST tests for all six generic index families: serial/parallel output comparisons, plan order, sparse and empty shards, incremental coverage, partition filtering, snapshot responses, typed options, invalid controls, internal/external private cleanup, and file retention after source-conflict commits.

The core performs scheduling and index construction. There is no Python retry after a native build failure.

Verification

• REST/native index regression suites: 253 passed, 85 actual native plans.
• New REST tests plus native integration gates with every Native CI switch enabled: 94 passed; 45 plans, 15 reads, 10 writes, 55 commits, and every tracked native update operation exercised.
• Existing Python global-index build suite: 38 passed, 19 subtests passed.
• Flake8 and Python 3.6 syntax checks for changed Python files: passed.
• Tests used an editable extension built from the paired Rust branch. No CI source pin or dependency was changed.

[leaves12138]

Reviewed 06e96806 against a freshly rebuilt apache/paimon-rust#1094 extension at 25183209. No blocking findings.

Resolving effective parallelism through the existing integer ConfigOption conversion preserves typed values, build-over-table precedence and explicit default resets before FFI. Full-text/vector builds remain native without a Python-row retry. Prepared messages retain plan order, selected-snapshot metadata, sparse source IDs and caller commit ownership; private failure cleanup and retained files after commit conflicts are preserved.

Independent local validation with Python 3.10 and PyArrow 19.0.1:

• Paired REST/index-build suites: 291 passed and 19 subtests passed, with 85 Native plans exercised.
• All five Native CI switches enabled: 94 targeted tests passed; 45 plans, 15 reads, 10 writes, 55 commits and every tracked native update operation kind were exercised.
• Additional reviewer tests: 40 passed for two simultaneous builds where one fails, internal/external output ownership, typed table-copy values and non-finite controls rejected before output.
• Rebuilt Python binding regression: 417 passed using a Native-generated smoke warehouse.
• Flake8 with dev/cfg.ini and Python 3.6 syntax parsing passed for changed Python files.

The paired suites exercised all six index families, serial/parallel metadata and deterministic vector bytes, both committers, partition selection, REST snapshot responses, incremental coverage and source-conflict file retention. Concurrent failure cleanup did not delete the other preparation's successful files.

Scope: Java alignment is source-level per-shard writer/snapshot/row-ID/ownership comparison, not Java-engine or production-storage interoperability. This local parameter retains the existing Python default of 1 and is distinct from Java's distributed global-index.build.max-parallelism control. Keep the Rust dependency sequencing: complete Native CI must use Rust main containing #1094; the current draft status is appropriate until that dependency is available. No production throughput benchmark was performed.