[HyukjinKwon]

What changes were proposed in this pull request?

SPARK-61146: PySpark Connect's createDataFrame of a pandas DataFrame under a struct schema whose structs repeat field names, such as struct<x: string, x: int>:

• converts the frame with the deduplicated names of _deduplicate_field_names (x_0, x_1)
• sends that Arrow table with the schema as given, which the server applies

The client converted the frame the same way, in prepare_pandas_table. But it then normalized the table again as an Arrow table of the caller (prepare_table), which may not repeat the field names of a struct. It now sends the converted table with the schema, as PySpark Connect does.

The second normalization changed nothing else for this table. Its types are already those of the schema, and its timestamps are already of UTC. It also fetched the Arrow input settings a second time, so createDataFrame of such a frame now sends one Config request fewer (integer_tuple_cases.py counts them).

Why are the changes needed?

The client raised:

UnsupportedOperationException: [DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT] Duplicated field names in Arrow Struct are not allowed, got ['x', 'x']

Apache Spark's test_createDataFrame_pandas_duplicate_field_names of test_parity_arrow.py failed on this client.

Does this PR introduce any user-facing change?

Yes. A pandas frame under a schema whose structs repeat field names becomes a DataFrame, as with PySpark 4.3.0.

How was this patch tested?

• A new offline test checks what is sent for such a frame: the Arrow table of the deduplicated names, of the values, and the schema as given. It fails without this change.
• A new live test creates DataFrames of such a frame, with Arrow and without, and checks their schemas and rows. It passes with official PySpark 4.3.0, and fails without this change.
• The schemas and rows of the case of Apache Spark's test, and of timestamps, at the top level and in a struct under a client time zone unlike the session's, are those of official PySpark 4.3.0.
• The lib tests of both crates (590 passed), the build of all tests, also with wasm-udf, the offline Python suites (2205 passed, 8 skipped), and the live drop-in and Arrow suites (188 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.