[dbtsai]

What changes were proposed in this pull request?

SPARK-60704: fix Decimal hook dispatch and conversion-cache lifetime in the conversion already on master. Nested pandas inference also relates to SPARK-60696.

• Resolve the canonical C Decimal class independently of a rebound public alias, caching both accelerator presence and absence. Exact C values retain the text shortcuts; subclasses and supported pure-Python values retain their hooks and keyword quantize(exp, context=...) call.
• Restore the outer quantum/context cache after reentrant conversion, including errors. Release cache borrows before invoking Python or dropping displaced Python objects; both quantize call forms share one context per Decimal type.
• On native pandas row fallback, widen inferred Decimal scales across observed nested array, struct and map values. Preserve container shape, trailing zeros and precision 38. Ordinary pandas inputs retain master's columnar conversion.

Why are the changes needed?

Subclass hooks can otherwise be bypassed or receive incompatible arguments, and reentry can discard the outer conversion's cache. First-value-only inference can reject a later Decimal("1.234") after Decimal("1.2"), although both fit at the later scale.

Does this PR introduce any user-facing change?

Yes. These are narrow correctness fixes with explicit compatibility limits:

• Class selection and precision loss: the effective class and accelerator availability are cached at first resolution; later alias or module-availability changes are not rediscovered. With C Decimal available, rebinding decimal.Decimal to _pydecimal.Decimal does not make pure-Python instances native Decimals. Local-row conversion/inference can fail; inferred or names-only native pandas fallback can instead coerce them through __float__ to doubles and lose precision. Quantums still use the selected class. Exact-C shortcuts assume CPython's immutable Decimal class.
• Context hooks: one context per Decimal type is created lazily at its first non-null, non-NaN value within a conversion. Reference PySpark constructs contexts per field converter before processing values. Hooks that alter context factories, DefaultContext or a supplied context can observe different timing and sharing. Changes between conversions apply to the next conversion.
• Nested scale, precision and errors: preserving the greatest observed scale can make previously accepted values exceed precision 38. Native fallback can raise PySparkRuntimeError during DataFrame construction before sending a relation; reference PySpark Connect 4.3.0 can construct that input and later raise ArithmeticException during an action. A newly increased scale above 38 raises ValueError locally. Existing explicit-nested conversion errors and Decimal-before-other-invalid-column ordering remain. Ordinary pandas columnar inputs retain Arrow's conversion errors.

This does not establish full Python behavioral parity or introduce a new scientific-notation parser.

How was this patch tested?

Candidate bbf493219fdda8bc668471163b936f34b5e46429 against baseline e8b1d4284e11d119c4fda9563a913ece22f43e7c:

• Focused native tests: 132 passed, one skipped. Full native suite: 2,260 passed, eight skipped, one failed. The sole failure matches the baseline's unsupported-NumPy-scalar case; the raw full-suite result remains failed.
• Independent full-patch source review found no actionable findings. Rust core/API/deadline checks and default/Wasm compilation passed in the exact-source focused receipt and are reused by the full-suite run. Paired binding Clippy evidence is carried through audited unchanged lint inputs: raw exit 101, zero new binding diagnostics. This is not clean final-head lint.
• Fresh official transport gate: 91 selected files, 2,601 passed, 655 runtime skips, zero unexpected failures/errors (87 local files: 2,586/641; four cluster files: 15/14). Eleven modules contain only skips, totaling 267 cases. A separately drift-run, checked-in excluded catalog case still fails because the distribution lacks InMemoryCatalog; it is outside those selected totals. Skips and exclusions are not passing coverage. Reference Python planning/conversion/decoding over Rust transport does not validate native Decimal conversion or full Python parity.
• The initial final-revision run completed all six measurement workers and contracts, then its large-upload live check exhausted the 1 GiB JVM heap; cleanup also raised PermissionError. The incomplete raw summary and failed-execution receipt are preserved; that run is not accepted as passed. A complete repeat uses an independently verified 3 GiB JVM heap, with unchanged sources, worker, driver and plan.
• The complete 3 GiB rerun passed: all six timing workers, output-contract checks and eight native live tests per candidate (including the >128 MiB upload), with no live skips or failures. The 12-case Decimal contract probe passed on the candidate and official PySpark; the baseline had three contract failures. All final source, import, binary and reference-input guards passed; the owned JVM was reaped.
• Published-head CI remains pending publication and exact-head verification.

Measurements use macOS, Spark Connect 4.3.0, pandas 2.3.3, PyArrow 26.0.0 and one shared owned local server. Decimal workloads have 200,000 rows and two fields; string workloads have three fields. Each worker is a separate process. The following values are client createDataFrame CPU / wall milliseconds. Complete output and schema validation ran outside the timer.

Warm medians of five calls:

Workload	Frozen master CPU / wall	Candidate CPU / wall	Pure Python CPU / wall
local-explicit	100.292 / 100.304	106.842 / 106.852	103.479 / 103.957
local-names	332.024 / 332.036	329.253 / 329.278	913.490 / 913.352
pandas-explicit	60.314 / 61.665	59.594 / 60.640	60.837 / 61.207
pandas-names	228.319 / 231.083	228.047 / 229.806	228.727 / 228.699

First measured call per workload: imports, session setup and input preparation precede these samples; later workloads follow earlier ones in their worker.

Workload	Frozen master CPU / wall	Candidate CPU / wall	Pure Python CPU / wall
local-explicit	105.641 / 105.667	106.004 / 106.014	118.921 / 165.571
local-names	331.917 / 331.925	330.871 / 330.879	925.916 / 956.206
pandas-explicit	76.151 / 139.416	73.075 / 115.458	61.859 / 62.985
pandas-names	224.299 / 263.237	217.553 / 281.154	227.704 / 267.435

Observed warm local-explicit CPU cost increased 6.5% versus master; the other three CPU medians changed by -1.2% to -0.1%. Whole-worker cumulative peak RSS was 273.1 MiB for master, 282.8 MiB for the candidate and 257.6 MiB for Python.

This is one fixed-order paired run without confidence intervals. RSS includes input preparation, expected tables, collection, validation and allocator retention. Guard-only cost, fallback/wide/nested workload performance, RPC/upload counts, conversion-only memory, process cold starts and whole-query performance are outside these measurements.

Was this patch authored or co-authored using generative AI tooling?

OpenAI Codex.

This pull request and its description were written by OpenAI Codex.