[dbtsai]

What changes were proposed in this pull request?

Preserve PySpark 4.3.0's timing for spark.sql.execution.pyspark.binaryAsBytes: eager actions read it after execution; toLocalIterator() snapshots it on the first next(), before execution. Apply it to Arrow-only results too.

Configuration or converter-initialization errors now terminate the iterator and release any incomplete execution. Closing an iterator also wakes prefetch while it waits for another response, allowing prompt execution release.

Why are the changes needed?

SPARK-61099, following initial binary support in #320. Reading configuration after the first fetch can consume a row before an error and allow later calls to resume. Schema-dependent configuration checks also miss Arrow-only results.

Includes the NumPy prerequisite, tracked by SPARK-61102.

Does this PR introduce any user-facing change?

Yes. Eager/lazy configuration timing and failed-iterator cleanup match PySpark. Direct binary map keys retain PySpark's TypeError when bytearray keys are unhashable. Binary-backed UDT map-key behavior remains outside this patch.

How was this patch tested?

• Formatting, Rust core/API units, integration-test compilation with and without WASM, and native-extension Python suites: 2,253 passed, eight live-server skips.
• Regression coverage includes eager aliases, both prefetch modes, absent/present Spark schemas, empty/null/nested values, configuration failures, GIL release and operation-matched execution release.
• Five native live cases and all 15 setup/call/teardown reports passed with actual imports verified; 179 unrelated cases were deselected. Each of three mock-RPC configuration-ordering tests passed in 100 rounds: 300 passing executions, with per-round import verification and process cleanup.
• Scoped official transport checks: 85 passed, 34 skipped, no retries. All 12 mixed Classic/Connect collection cases skipped. API inspection found no missing inspected members across 12 surfaces; it does not establish complete signature or behavior parity.

The shared full and cluster runs executed the NumPy prerequisite binary. This branch has the same 43 selected raw-RPC source files and separate native-import/live validation. This is source-based reuse of transport evidence; those full and cluster runs did not execute this branch's binary. They use official Python plans and converters, with no per-child import attestation, and do not validate native high-level conversion, native ReleaseSession or native artifact upload.

The full run completed 87 normal files: 2,586 passed, 641 skipped, one deselected and no failures, errors or retries. Eleven files were entirely skipped. A separate advisory check of the existing metadata-column manifest entry still failed because the test catalog plugin was unavailable (one failure, 57 deselections). The cluster phase passed 15 invocations across four modules (ten substantive cases and five helpers), with 14 JVM-guarded skips and no failures or retries.

The original full run had a seeded-random failure on two threads. The unchanged test failed identically in official Python and both saved Rust transports on two threads, and passed in all three on its default four-thread fixture. The complete successful rerun used that fixture. Server logs retain deliberate negative-test diagnostics and shutdown races; successful direct shutdown was recorded, without final descendant-process telemetry.

Clippy remains advisory exit 101; its source diagnostics were assessed against the baseline. Linux, Windows and the newest-master combined tree were not runtime tested locally.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex

This pull request and its description were written by OpenAI Codex.