[dbtsai]

Current change after the master rebase

Encode eligible pandas NumPy object-string columns directly into Arrow IPC in Rust for frames of at least 10,000 rows with an explicit native UTF8_BINARY string schema and exact str/nullable None cells. Preserve inline LocalRelation transport, Spark schema and supported fallback/error ordering for subclasses, surrogates, missing values and customized hooks. This remains stacked on PR #121 with current Index-iteration coverage and its inline bridge; PR #118/#108 are already in master. Guards cover a finite set of directly skipped hooks rather than arbitrary pandas customization.

Validation of the rebased change

Validated exact head e31aad6482c1b748924991a8c01aabbea74db385 against upstream master 014674e44470001f5c184846964c131d43f6e4fa. Formatting, 115 core and 419 API Rust unit tests, all API integration-target compilation with and without wasm-udf, the native extension build, and 979 offline/mock Python tests passed (8 skips). Committed whitespace checks passed. Workspace Clippy remains advisory and fails on unmodified master; diagnostics are compared separately in the rebase evidence. Full live-Spark parity and CI for the rewritten head remain pending. No new performance, memory, or RPC-cost measurement is claimed; historical results do not validate this rebased head.

Historical pre-rebase description, validation and performance evidence

The complete previous description is preserved below as historical evidence. Every test count, benchmark number, source/dependency/revision ID, RPC observation, source-equivalence claim, and use of “current”, “final”, “fresh” or “new” in this appendix refers to its original pre-rebase context. Master has since changed local inference and conversion, cached schema access and Arrow configuration handling. The current scope above takes precedence over stale runtime, dependency and per-call RPC statements below. None of these results validates or measures the November 5 rebased head. Prior failures, guard/cold-path costs, RPC/memory observations and live-server limitations remain preserved within their stated historical scope.

What changes were proposed in this pull request?

Pandas object columns containing strings currently materialize Python rows during createDataFrame, even with an explicit string schema. Encode eligible columns directly into Arrow IPC in Rust while preserving the existing inline LocalRelation transport and Spark schema.

The path requires an ordinary pandas DataFrame with at least 10,000 rows, concrete NumPy object storage, an explicit native UTF8_BINARY string schema, and exact str values or nullable None. Customized pandas/schema hooks, string subclasses, unsupported missing values and surrogates retain the existing conversion/error path. The required internal IPC bridge is included in this change.

Depends on PR121, including its PR118/PR108 prerequisites. The narrow delta against PR121 changes five production files and adds 19 test definitions.

Why are the changes needed?

The frozen client-only benchmark compares published PR121 c4284a3e0b71c8d624a1253c52337419761b2131 without the bridge against candidate d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723 with the complete fix, using separate fresh wheels/environments and unchanged workloads/validators. All 535 samples completed. An independent passive raw artifact audit cross-checks original validator receipts, hashes and all raw metrics; it does not replay Arrow decoding.

Medians below use five worker samples. Each string frame has 10,000 rows and three columns. Each focused warm sample averages 20 measured constructions after three warmups; focused first-construction samples measure one construction after zero warmups.

Case/profile	Baseline ms	Candidate ms	Change
Ordinary pandas 2.3.3 object strings, warm	23.137	2.154	−90.7%
Ordinary pandas 2.3.3 object strings, first construction	24.298	2.935	−87.9%
Focused all-null strings, warm	2.868	0.876	−69.5%
Late subclass fallback, first construction	23.938	25.921	+8.3%
Early surrogate error, warm	1.415	1.659	+17.3%
Middle surrogate error, first construction	13.453	15.252	+13.4%

The middle-surrogate candidate maximum was 21.036 ms versus 13.605 ms baseline. The ordinary pandas 3.1 StringDtype case retains the fallback path and increased 1.95% warm/6.61% first-construction elapsed time. Ordinary Arrow input increased 15.0% in warm elapsed time while CPU decreased 8.4%. Other guarded workloads increased: the million-row pandas float case by 3.93% elapsed/5.04% CPU and UDF construction by 4.27% elapsed/3.92% CPU. Focused CPU medians increased 18.72% for early-surrogate warm errors, 12.17% for middle-surrogate first-construction errors and 7.75% for late-subclass first construction. These five-sample observations do not establish statistical significance or a general overhead bound.

Warm ordinary held-process RSS under pandas 2.3.3 was:

Input	Baseline MiB	Candidate MiB
Pandas object strings, 10,000 × 3	113.672	107.000
Pandas array values, 10,000 × 3	121.703	121.047
Wide pandas input, 10,000 × 32	118.328	118.953

Measured construction starts after interpreter/library/session/input preparation. Ordinary first-construction timing also follows an untimed version call; the focused worker has no explicit version call. First-construction samples measure one constructor call without constructor warmup. Cold-interpreter costs were not measured. Focused sampled-peak/post-GC RSS was not collected; ordinary post-GC RSS includes later correctness work.

The client is 4.3.0 on Python 3.12.14 with pandas 2.3.3/3.1.0, NumPy 2.5.4, PyArrow 26.0.0 and grpcio 1.84.1. A separate official PySpark 5.0.0.dev0 reference at Apache Spark ca1514a2b4e72f9529871f34d09881cd4921df82 is measured under pandas 2.3.3 only. The Rust-client and official PySpark versions differ, so the official result provides separate context. Improvements above compare the two Rust-client revisions; historical minimal/combined-tree timings were not substituted.

Does this PR introduce any user-facing change?

Eligible pandas string input uses the column encoder. Tests preserve values, nulls, field names/metadata, error ordering and inline transport/configuration behavior. Paired ordinary 20-counter RPC maps are equal. Focused intercepted RPC method maps are zero; their separate inferred zero 20-counter maps do not establish a complete end-to-end transport observation.

How was this patch tested?

• Both exact revisions pass formatting and release library tests: 394 client tests and 95 core tests each. Native and host wasm-udf feature test compilation each produce 37 test executables; no WASM-target execution is claimed.
• Fresh wheels pass complete same-source test_arrow_handoff.py and test_dropin_offline.py using pytest --rootdir=python --import-mode=importlib -q -rs. Baseline: 522 passed/9 skipped for each pandas version. Candidate: 681 passed/9 skipped with pandas 2.3.3; 679 passed/11 skipped with pandas 3.1.0. The two extra skips concern unavailable older infer-string storage behavior.
• All 531 baseline collected cases/outcomes are preserved. The 19 added definitions cover 159 cases per pandas version, including complete IPC cells/schema, subclass/surrogate fallback, descriptors/aliases, normalization and backing-array hooks, column storage and large inline input.
• An independent passive raw artifact audit cross-checks original validator receipts/hashes and all raw metrics for all 535 samples: 430 IPC, 60 expected-error and 45 guard samples, with raw requests/IPC, distributions and adverse costs retained. It does not replay Arrow decoding. All six supervised process groups are absent before releasing the shared lock; four actual role packages and the separate official reference retain their required identities.

Package checks cover existing dependency RECORD-listed non-bytecode files and recorded PySpark import origins; they do not establish continuous isolation, unlisted/shadow dependencies or loaded dependency origins. Probe receipts cover outer role/reference commands; nested internal check_output partial stdout is not separately receipted.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

This pull request and its description were written by Isaac.

[HyukjinKwon]

Thank you for this. As agreed with the maintainers, it continues in #208: this branch does not allow edits by maintainers, so #208 carries its commits plus the merge with master (which now has #121) and with #207, the replacement of its sibling #143, as the two share the inline IPC bridge and the dispatcher, and the review changes (encoding each string once instead of twice, dropping the converter checks the string path never reaches with the tests that only exercised them, and relaxing the test that pinned master's pandas read counts). It keeps SPARK-60720. #145, stacked on this, stays open.