[HyukjinKwon]

What changes were proposed in this pull request?

SPARK-61177: StructType.toDDL() quotes each field name with quoteIfNeeded, which wraps the name in backticks, doubles the backticks in it, and leaves a name alone only when it matches [a-zA-Z_][a-zA-Z0-9_]*. The types are written as DataType.sql, so a nested struct is STRUCT<`my col`: INT NOT NULL COMMENT 'c'>. DataType.fromDDL canonicalizes the collation name, as the server does.

The Rust client now does the same in DataType::to_ddl and in its from_ddl parser:

• to_ddl quotes the top-level names and the names of nested struct fields. It writes the types in the DataType.sql form: STRUCT<...>, ARRAY<...>, MAP<K, V>, and STRING COLLATE <NAME> for a non-default collation.

• from_ddl takes the collation name from the input, not from the lowercased copy, and canonicalizes it with canonical_collation_name. A UTF8_* or UNICODE* name is in upper case. An ICU locale name, language[_Script][_COUNTRY][_SPECIFIERS], has a lower-case language, a title-case four-letter script and upper-case other parts. The server gives these names for the input in the first column:

  written	canonical
  utf8_lcase	UTF8_LCASE
  Unicode_Ci_Ai	UNICODE_CI_AI
  unicode_rtrim	UNICODE_RTRIM
  utf8_lcase_rtrim	UTF8_LCASE_RTRIM
  sr_ci_ai, SR_CI_AI, sr_CI_AI	sr_CI_AI
  EN	en
  de_ci	de_CI
  zh_hant_hkg	zh_Hant_HKG
  SR_CYRL_SRB_CI_AI	sr_Cyrl_SRB_CI_AI

  The server rejects en_us with COLLATION_INVALID_NAME. from_ddl does not validate names, so it gives en_US.

Why are the changes needed?

With the old code, to_ddl wrote the names as they were. The server cannot parse the result for a name that needs quoting:

createDataFrame([], "my col int")
  ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near 'int': extra input 'int'.
createDataFrame([], "a`b int")
  ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near '`'.
createDataFrame([], "s struct<my col:int>")
  ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near '<'.

The strings are what the old to_ddl wrote for a struct with a field my col of type INT, a field a`b of type INT, and a struct field s of type struct<my col:int>.

from_ddl("a STRING COLLATE UTF8_LCASE") gave the collation utf8_lcase. Spark gives UTF8_LCASE.

Does this PR introduce any user-facing change?

Yes. DataType::to_ddl now returns the Spark DDL form, for example `my col` INT,`ab INT `` in place of ``my col int,ab int``. A nested struct, array or map is written with the SQL type names. DataType::from_ddl canonicalizes the collation name, for example `UTF8_LCASE` for `STRING COLLATE utf8_lcase` and `sr_CI_AI` for `STRING COLLATE SR_CI_AI`, as the server does.

How was this patch tested?

• struct_to_ddl_quotes_names_and_writes_sql_types in crates/spark-connect/tests/types_behavior.rs covers quoted and unquoted names (spaces, backticks, dots, leading digits, all digits, leading underscore, unicode), NOT NULL, comments with a single quote, nested struct fields, arrays and maps of structs, atomic types, and collations. The expected strings are the output of StructType.toDDL() on the server of PySpark 4.3.0. This test fails without the change.
• from_ddl_canonicalizes_the_collation_name in crates/spark-connect/tests/types_behavior.rs checks the twelve names of the table above, with the canonical name the server gives for each (en_us as en_US), and a to_ddl round trip. It fails without the change (utf8_lcase before the change).
• struct_to_ddl_tree_and_nullable was updated to the new output a INT,b STRING NOT NULL.
• Against the server of PySpark 4.3.0, createDataFrame([], "c STRING COLLATE <name>") gives the canonical names of the table above, which the new test checks.
• Against the server of PySpark 4.3.0, the DDL strings written by to_ddl parse back (createDataFrame([], ddl).schema) to the expected types, for the names, the nested structs, the arrays and maps, and the collations in the cases above. The CHAR and VARCHAR case fails in createDataFrame with an internal check on the local relation. This is not a parse error.
• The lib tests of both crates (608 passed), the build of all tests, also with wasm-udf, the offline Python suites (2227 passed, 8 skipped), and the live drop-in and Arrow suites (210 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.