[zhengruifeng]

What changes were proposed in this pull request?

Add native LinearRegression and its fitted model so Rust callers can train Spark's JVM regressor and predict on new DataFrames through Spark Connect. Fit sends the dataset and parameters through MlCommand.Fit; transform uses the returned cached model reference and the existing model ownership/session handling.

Expose typed Rust setters for all 15 constructor parameters, including elastic-net regularization, instance weights, solver selection and squared-error/Huber loss. Add a Python binding under pyspark.ml.connect.regression.LinearRegression and pyspark.ml.connect.LinearRegression, with the reference keyword-only constructor and mutating parameter setters. Encode the two JVM integer parameters as 32-bit literals to match PySpark.

Why are the changes needed?

The native Rust API currently has no linear regression estimator. This adds regression alongside LogisticRegression and LinearSVC, using the existing ML transport and RegressionEvaluator.

SPARK-61193

Does this PR introduce any user-facing change?

Yes. Rust callers and users of the native Python binding can fit a regressor with vector features and numeric labels, then transform DataFrames in its fitting session. Models, clones and lazy transformed DataFrames retain the fitted server object until their last owner is dropped. Invalid training data or parameters fail during fit.

The native API added here covers fitting and DataFrame transformation. Coefficients/intercept access, summaries, single-vector prediction and persistence are not exposed by this addition; these limits are documented. The existing vendored pyspark.ml.regression module is unchanged.

How was this patch tested?

• Added a golden comparison of all 15 parameter encodings against official PySpark 4.3.0, with a capture script for regenerating the fixture. Added offline validation of fitted model references and metadata.
• Four live Rust tests passed against Spark 4.3.0: held-out predictions for known lines and RMSE < 1e-6, independent fitted datasets, cloning/lazy execution, session isolation, default/custom columns, zero-weight outliers, iterative elastic-net regularization, Huber loss and invalid data/parameters.
• Three Python cases passed with both the native binding and official JVM PySpark as reference. Seven selected native Python integration tests passed across LinearRegression, LogisticRegression and LinearSVC. For the reference-only setter case, loss is set through setParams: upstream v4.3.0 setLoss incorrectly targets the nonexistent lossType parameter. The native setLoss targets loss.
• Rust full test run without SPARK_REMOTE: 741 passed, 26 ignored; server-gated test bodies return early without a server. All integration targets compile with and without wasm-udf; formatting, library Clippy and Clippy on the two new integration/golden test targets passed.
• Python offline suite: 1,230 passed, 8 skipped. Verified the new constructor's 15 names, defaults and keyword-only arguments against PySpark; the existing 12 core wrapper classes have no method-name gaps under the documented transport exclusions.
• Workspace/all-target Clippy reports existing errors. Comparing diagnostics with the exact base commit found no new errors.
• Official SQL Connect suite through the Rust transport: all 87 files passed (2,638 tests passed, 589 skipped), using the existing CI exclusions and known-failure manifest.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-6)

This pull request and its description were written by Isaac.