[HyukjinKwon]

What changes were proposed in this pull request?

SPARK-61178: PySpark's StructType holds its fields in a list attribute, fields, and gives back the same list on each access. Its names is a separate list, which __init__ and add set. The attributes name, dataType, nullable and metadata of StructField are plain attributes.

The Python client now mirrors this:

• StructType holds its fields (a list of StructField objects) and its names (a list of strings) as Python lists. The struct reads its fields from them each time, so append, item assignment, pop, sort and del on fields change the struct: simpleString(), json(), len, iteration and indexing. A list kept before add() sees the new field.
• add() appends to both lists. fieldNames() returns a copy of names, so it does not follow an edit of fields. __eq__ also compares names.
• StructField.name, dataType, nullable and metadata are assignable. An assignment is seen by the struct the field is in.
• StructField.dataType, ArrayType.elementType, MapType.keyType and MapType.valueType keep their Python object: the object given to the constructor or setter, or one made from the core type on first access. So they are the same object each time, and an edit of a nested type (in place, or through its parent) is seen by the outer type, in its json, simpleString, equality, hash, pickling, createDataFrame schemas and UDF return types.
• _needConversion and _needSerializeAnyField are computed by __init__ and add, as in PySpark, so an edit of a field's type does not change them.
• The core type that the Rust side reads (for createDataFrame schemas, UDF return types, json, equality, toDDL) is built from the live lists.

Not mirrored: after a field is appended to fields (not with add()), createDataFrame on local data builds the rows from all the fields. PySpark raises there, because its row conversion uses names. Pickling a struct keeps only the names of its fields, so a struct whose names disagree with its fields unpickles with the names of the fields.

Why are the changes needed?

Each access to fields built a new list, and the StructField attributes were read-only. So edits in place were lost, and the common way to change nullability did not work:

>>> st = StructType([StructField("a", IntegerType())])
>>> st.fields.append(StructField("b", StringType()))
>>> st.simpleString()          # before this change
'struct<a:int>'               # PySpark: 'struct<a:int,b:string>'
>>> for f in st.fields: f.nullable = False
AttributeError: attribute 'nullable' of 'pyspark.sql.types.StructField' objects is not writable

Does this PR introduce any user-facing change?

Yes. StructType.fields is the same list on each access, and edits of it or of a StructField (name, dataType, nullable, metadata) change the struct, as in PySpark. fieldNames() and names keep the names that __init__ and add set, so they can disagree with fields after a list edit, as in PySpark.

How was this patch tested?

• New offline tests in python/tests/test_dropin_offline.py: test_struct_type_fields_is_the_live_list_of_the_struct, test_struct_type_add_keeps_the_fields_list, test_struct_type_names_are_apart_from_fields, test_struct_field_attributes_are_assignable, test_struct_type_edits_reach_its_core_type, test_struct_type_nested_types_are_kept_as_pyspark_keeps_them. The first five fail on the code before this change, and pass with it. The same six tests pass unchanged against official PySpark 4.3.0. The nested test was not run on the build before this change. Its identity checks (dataType is dataType) were False on that build in the coordinator's probe.
• New e2e test in python/tests/test_dropin_e2e.py: test_create_dataframe_with_a_schema_edited_in_place. It edits nullability in place and adds a field with add(), and checks the schema and rows of createDataFrame. Official PySpark 4.3.0 gives the same schema and rows against the same server.
• The existing -k "struct or schema or types" offline tests and the matching e2e tests pass.
• The lib tests of both crates (608 passed), the build of all tests, also with wasm-udf, the offline Python suites (2233 passed, 8 skipped), and the live drop-in and Arrow suites (211 passed) and live cases (11 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.