[HyukjinKwon]

What changes were proposed in this pull request?

SPARK-61179: PySpark's DataFrameReader.option(key, None), DataFrameReader.options(key=None) and DataFrameReader.load(..., key=None) unset the option, and Catalog.createExternalTable(..., description=...) passes the description to createTable, which stores it as the table comment.

• DataFrameReader.option, options and load unset an option given None, through a new core DataFrameReader::unset_option. The format methods (csv(path, header=None) and the like) still skip None and keep the earlier value, as PySpark does, and so do the DataStreamReader and writer option methods, which PySpark also keeps.
• Catalog.createExternalTable takes description as a named argument and passes it to createTable, instead of sending it as a table option.

Why are the changes needed?

A reader option set to None kept its earlier value, so the reader read with the header that PySpark drops:

spark.read.option("header", "true").option("header", None).format("csv").load(path).columns
# PySpark 4.3.0: ['_c0', '_c1']
# before this change: ['a', 'b']

createExternalTable with a description did not set the table comment:

spark.catalog.createExternalTable("t", path=p, source="parquet", description="the table")
spark.catalog.getTable("t").description
# PySpark 4.3.0: 'the table'
# before this change: None (and SHOW TBLPROPERTIES listed option.description)

Does this PR introduce any user-facing change?

Yes. DataFrameReader.option, options and load with a None value now unset the option, as in PySpark. Catalog.createExternalTable(description=...) sets the table comment, as in PySpark, and no longer adds a table option.

One difference remains: PySpark's DataFrameReader._options lists an unset key with the value None, and the Rust-backed reader drops the key.

How was this patch tested?

• test_reader_none_unsets_an_option_and_external_description_is_the_comment in python/tests/test_dropin_e2e.py. It checks the reader columns after option, options and load with None after an earlier header value, that the csv(path, header=None) kwarg keeps the earlier value, and that a reused reader keeps its option, and it checks getTable(...).description, the Comment row of DESCRIBE TABLE EXTENDED, and that SHOW TBLPROPERTIES lists no description. It fails without this change, at the first option case (['a', 'b'] for ['_c0', '_c1']). The same test passes with official PySpark 4.3.0 over sc://.
• The same catalog and DESCRIBE results were checked with official PySpark 4.3.0, this change and master by a probe script.
• option(key, None) keeps the earlier value in official PySpark 4.3.0 for DataFrameWriter, DataFrameWriterV2 and DataStreamReader (probed through _write and _options), and on the build for DataFrameWriter (by reading back the written header) and DataStreamReader (through _options). The build's DataFrameWriterV2 path is unchanged and was not probed.
• The lib tests of both crates (608 passed), the build of all tests, also with wasm-udf, the offline Python suites (2227 passed, 8 skipped), and the live drop-in and Arrow suites (211 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.