[python] Support distributed vector index construction with Ray

[TheR1sing3un]

Purpose

Allow create_global_index and multimodal create_index to build native vector-index shards with execution="ray". The driver plans uncovered row ranges from one snapshot, workers construct shards with bounded concurrency, and the driver publishes all resulting entries in one commit. Local execution remains the default.

Allocate output names before dispatch and disable automatic write-task retries. If a task fails, drain outstanding tasks before removing uncommitted shard files, including files whose commit messages were never returned. Document shared storage, dependency, snapshot-retention and driver-failure requirements.

Tests

50 targeted tests passed with Ray 2.54.0 and paimon-vindex 0.5.0, covering distributed construction, incremental coverage, deleted rows, one-commit publication, build/commit separation, cleanup after a worker writes then fails, option validation, and existing global-index/multimodal APIs. Flake8 and git diff --check passed. No benchmark code included.

Review follow-up: JindoFileSystemHandler now serializes only its root and connection options, reconstructing the native client, lock and size-hint cache during deserialization. Tests cover an unpicklable SDK client, a PyArrow filesystem round trip, and the complete builder/TableRead context crossing into a real Ray worker (only SDK Config/connect are substituted). The previous implementation fails the new pickle regression. Validation: 21 Jindo/vector-build tests passed on Ray 2.54.0; 19 real-SDK tests skipped because pyjindosdk is unavailable locally. The worker-context regression also passed on Ray 2.59.0.

[JingsongLi]

+1