[dbtsai]

What changes were proposed in this pull request?

SPARK-60745: preserve native row-conversion behavior for customized Arrow-backed pandas strings in the columnar input path already on master.

Specialized, generic explicit-schema and inferred/names-only routes share the relevant hook and alias eligibility checks. Covered custom array/label conversion hooks and rebound schema-converter aliases decline to native rows, so another admission route cannot bypass those rejections. The specialized string helper separately declines recognized storage-normalization and validation failures, including a column storage length that differs from the frame. Ordinary supported inputs retain the existing columnar paths and size/schema thresholds.

Inspect current instance hooks and selected canonical class attributes, including inherited methods and pandas' NumPy-semantics mixin/partial dispatch. Read raw class attributes without binding customized descriptors, and recheck between columns because an accessor can change a hook. Only the pandas helper, regression tests and architecture documentation change.

Why are the changes needed?

Direct Arrow-storage extraction can bypass a customized conversion method, change values, suppress an exception or run label callbacks earlier than native row conversion. The covered fixtures must retain native values, errors and ordering through every applicable admission route.

Does this PR introduce any user-facing change?

Yes. Inputs with the covered customizations fall back to native rows; ordinary eligible inputs retain the existing fast path. Authentication adds per-column work, and conservative fallback can cost more than columnar conversion.

The guarantee is bounded to the selected canonical Arrow-string classes, instance/class hooks and explicitly checked aliases. Class data read by those methods, including _storage and _na_value, and other module globals are outside the checks. Other array classes retain the existing generic-route module-origin policy. Arbitrary private storage replacement or inconsistent frame internals are not covered by an all-route consistency guarantee. This does not establish an invariant for arbitrary pandas mutations or full Python behavioral parity. A guard can already have read frame.dtypes before inspecting the array; it is not a promise that pandas callbacks never run.

Existing importer costs remain: configuration RPCs, validation/casting, alignment, IPC serialization and threshold-triggered artifact uploads. Narrowing large_string to Arrow string can copy the offset buffer; this path is not universally zero-copy. Simulated cast failures do not establish support for actual storage above 2 GiB or malformed offsets. The inherited package/runtime pandas-minimum mismatch and support for older/broken dependencies are not repaired here.

How was this patch tested?

Candidate 33559dc2af3df57b4db56b1cb90f16d1f5bbd8c5 against baseline e8b1d4284e11d119c4fda9563a913ece22f43e7c:

• Independent full-patch source review found no actionable findings. 216 focused native cases passed. Full native suite: 2,378 passed, eight skipped, one failed; the sole failure matches the baseline's unsupported-NumPy-scalar case. The raw full result remains failed.
• Version matrix: 216 passed in each of four environments—pandas/PyArrow 2.2.3/18, 2.3.3/26, 3.0.6/26 and 3.1.0/26. Actual imports, source and binary hashes were checked. Controls produced 132 candidate passes and 132 expected assertion failures: 120 on baseline and 12 with intentionally memoized guards. Wrong-origin failures were not accepted. Simulated cast failures remain distinct from real capacity/offset coverage.
• Rust core/API/deadline checks and default/Wasm compilation are carried from the fresh baseline through equality of all 190 native build inputs. The native Python tests imported the recorded baseline extension; no fresh Arrow Rust build is claimed. Baseline workspace Clippy remains a disclosed nonzero failure. A 12-class name audit found zero missing reference names and five extra native names; it does not certify signatures, behavior or full API parity.
• The completed fresh official transport gate has 91 selected files, 2,601 passed and 655 runtime skips, with zero unexpected selected-suite failures/errors. Eleven all-skipped modules account for 267 skips. A separately run, checked-in excluded catalog case fails because the distribution lacks InMemoryCatalog. A final 48-input transport/build/harness equality audit supports carrying this result to Arrow. Arrow's binary did not execute this official suite; it used the Decimal binary with reference Python planning/conversion/decoding. This is not native Arrow conversion validation or full Python parity.
• The initial final-revision run completed all six measurement workers and contracts, then its large-upload live check exhausted the 1 GiB JVM heap; cleanup also raised PermissionError. The incomplete raw summary and failed-execution receipt are preserved; that run is not accepted as passed. A complete repeat uses an independently verified 3 GiB JVM heap, with unchanged sources, worker, driver and plan.
• The complete 3 GiB rerun passed: all six timing workers, output-contract checks and eight native live tests per candidate (including the >128 MiB upload), with no live skips or failures. All six inferred/names-only Arrow-string contract cases passed on the baseline, candidate and official PySpark, with full values and schemas checked. All final source, import, binary and reference-input guards passed; the owned JVM was reaped.
• Published-head CI remains pending publication and exact-head verification.

Measurements use macOS, Spark Connect 4.3.0, pandas 2.3.3, PyArrow 26.0.0 and one shared owned local server. Decimal workloads have 200,000 rows and two fields; string workloads have three fields. Each worker is a separate process. The following values are client createDataFrame CPU / wall milliseconds. Complete output and schema validation ran outside the timer.

Warm medians of five calls:

Workload	Frozen master CPU / wall	Candidate CPU / wall	Pure Python CPU / wall
arrow-strings-1000	0.844 / 1.355	0.888 / 1.377	0.768 / 0.784
arrow-strings-200000	5.823 / 6.501	5.915 / 6.504	4.586 / 5.180

First measured call per workload: imports, session setup and input preparation precede these samples; later workloads follow earlier ones in their worker.

Workload	Frozen master CPU / wall	Candidate CPU / wall	Pure Python CPU / wall
arrow-strings-1000	9.802 / 10.918	11.856 / 12.689	1.257 / 1.820
arrow-strings-200000	10.882 / 37.037	10.735 / 42.028	13.557 / 39.020

Observed warm CPU cost increased 5.2% at 1,000 rows and 1.6% at 200,000 rows versus master; warm wall time changed by +1.6% and approximately 0.0%. The candidate remains slower than Python on these two workloads; the already-landed fast path is part of the baseline. Whole-worker cumulative peak RSS was 163.1 MiB for master, 165.4 MiB for the candidate and 174.9 MiB for Python.

This is one fixed-order paired run without confidence intervals. RSS includes input preparation, expected tables, collection, validation and allocator retention. Guard-only cost, fallback/wide/nested workload performance, RPC/upload counts, conversion-only memory, process cold starts and whole-query performance are outside these measurements.

Was this patch authored or co-authored using generative AI tooling?

OpenAI Codex.

This pull request and its description were written by OpenAI Codex.