Repository navigation

[SPARK-58987][INFRA][PYTHON] Faster parity CI, Rust coverage gate, and drop-in pyspark API fidelity - #54

Closed

HyukjinKwon wants to merge 100 commits into

apache:masterfrom

HyukjinKwon:better-parity-and-coverage

Closed

[SPARK-58987][INFRA][PYTHON] Faster parity CI, Rust coverage gate, and drop-in pyspark API fidelity#54
HyukjinKwon wants to merge 100 commits into
apache:masterfrom
HyukjinKwon:better-parity-and-coverage

Commits on Aug 25, 2026
- [SPARK-58987][INFRA] Parity gate: run official suite through our client only
  
  HyukjinKwon
  and
  Isaac
  committed
  20af6e1
- [SPARK-58987][INFRA] Add Rust coverage CI gate + fill retries/storage gaps
  
  HyukjinKwon
  and
  Isaac
  committed
  58858ea
- [SPARK-58987][INFRA] Add the generated known-environmental-failures manifest
  
  HyukjinKwon
  and
  Isaac
  committed
  54c299b
- [SPARK-58987][INFRA] Parity gate: retry flaky files; coverage: match ext arch
  
  HyukjinKwon
  and
  Isaac
  committed
  eed74ad
- [SPARK-58987][INFRA] Coverage: target 90%; pin one arch for local merged runs
  
  HyukjinKwon
  and
  Isaac
  committed
  d1f89c3
- [SPARK-58987][INFRA] Fix off-by-one bound in the jitter retry test
  
  HyukjinKwon
  and
  Isaac
  committed
  9cc44c9
- [SPARK-58987][INFRA] Lint: fix ruff E741/B007 and rustfmt in new code
  
  HyukjinKwon
  and
  Isaac
  committed
  0990f32
- [SPARK-58998][PYTHON] Restore drop-in API fidelity: Column/DataFrame names
  
  HyukjinKwon
  and
  Isaac
  committed
  035b4e3
- [SPARK-58987][INFRA] Keep coverage as tooling; defer the gating CI job
  
  HyukjinKwon
  and
  Isaac
  committed
  750abe2
- [SPARK-58998][PYTHON] Drop-in fidelity: DataFrame aggregation/util methods
  
  HyukjinKwon
  and
  Isaac
  committed
  046314b
- [SPARK-58998][PYTHON] Drop-in fidelity: spark.conf/read/catalog + df.stat
  
  HyukjinKwon
  and
  Isaac
  committed
  6f1bb26
- [SPARK-58998][PYTHON] Drop-in fidelity: filter/where(str), pivot, unpivot, more
  
  HyukjinKwon
  and
  Isaac
  committed
  f6d3af8
- [SPARK-58998][PYTHON] Drop-in fidelity: writer.option accepts non-str values
  
  HyukjinKwon
  and
  Isaac
  committed
  b3939b0
- [SPARK-58998] Fix Sort plan dropping bare (non-SortOrder) columns
  
  HyukjinKwon
  and
  Isaac
  committed
  d3cea30
- [SPARK-58998] Close the drop-in API gap; complete Arrow type mapping
  
  HyukjinKwon
  and
  Isaac
  committed
  2ab6e75
- [SPARK-58987][INFRA] ruff format the scripts
  
  HyukjinKwon
  and
  Isaac
  committed
  d5624d1
- [SPARK-58998][PYTHON] Skip window_with_slide_and_start in the dispatch generator
  
  HyukjinKwon
  and
  Isaac
  committed
  b64da1b
- Fix streaming query command results being discarded (status/isActive/manager)
  
  HyukjinKwon
  and
  Isaac
  committed
  ca32e8c
- Add coverage smoke tests for every function/plan/DataType builder
  
  HyukjinKwon
  and
  Isaac
  committed
  762f259
- Fix 13 variadic functions silently dropping all arguments
  
  HyukjinKwon
  and
  Isaac
  committed
  325f990
- Fix parity gate: manifest deselection never matched (gate ran every skip)
  
  HyukjinKwon
  and
  Isaac
  committed
  bc33579
- Make parity deselection rootdir-independent (plugin suffix match)
  
  HyukjinKwon
  and
  Isaac
  committed
  67a9c99
- Address review: Sort enum constants, GIL detach, map keys, conversion tests
  
  HyukjinKwon
  and
  Isaac
  committed
  931fbcb
- Address review: harden the parity gate (retries, drift check, auditable manifest)
  
  HyukjinKwon
  and
  Isaac
  committed
  60071a6
- Address review: fix non-compiling doc snippets + add docs CI validation
  
  HyukjinKwon
  and
  Isaac
  committed
  1364558
- Address review: rust_coverage.sh must fail if the Python suite can't run
  
  HyukjinKwon
  and
  Isaac
  committed
  f352fd2
- Address review: assert values in e2e_wrapper, not just absence of exceptions
  
  HyukjinKwon
  and
  Isaac
  committed
  964e936
- Build the crate before the docs snippet check
  
  HyukjinKwon
  and
  Isaac
  committed
  9ce170e
- Drop-in fidelity: options(**kwargs), StreamingQuery.explain() default; wider e2e
  
  HyukjinKwon
  and
  Isaac
  committed
  9585141
- Trim parity manifest to CI reality (drift check found 29 stale skips)
  
  HyukjinKwon
  and
  Isaac
  committed
  4a4d5c4
- Add native Rust e2e coverage for tvf/catalog/session/group/streaming/merge
  
  HyukjinKwon
  and
  Isaac
  committed
  544f2c1
- Cover read/write round-trips (readwriter.rs) and Row/Value accessors
  
  HyukjinKwon
  and
  Isaac
  committed
  cdfa2f9
- Cover the artifact-upload and tag APIs (important client-side paths)
  
  HyukjinKwon
  and
  Isaac
  committed
  f194380
- Cover the client-side ML request-building (ml.rs 62%->79%)
  
  HyukjinKwon
  and
  Isaac
  committed
  400c747
- Drop-in fidelity: addArtifacts(*paths) varargs; test artifact upload in Python e2e
  
  HyukjinKwon
  and
  Isaac
  committed
  5a894a1
- Replace proto-snapshot coverage tests with behavioral Column e2e
  
  HyukjinKwon
  and
  Isaac
  committed
  9654b9f
Commits on Aug 26, 2026
- Fix select_expr and to_json (found by behavioral e2e), add DataFrame e2e
  
  HyukjinKwon
  and
  Isaac
  committed
  600c8e4
- Fix DataType::String.simpleString for the default collation (behavioral test)
  
  HyukjinKwon
  and
  Isaac
  committed
  37c442c
- Fix parity gate holes: score on pytest returncode, match plural "errors"
  
  HyukjinKwon
  and
  Isaac
  committed
  9681ace
- Address review: createDataFrame TIMESTAMP regression + 6 drop-in fidelity fixes
  
  HyukjinKwon
  and
  Isaac
  committed
  7525aca
- Extend behavioral type tests: proto/JSON round-trips + StructType helpers
  
  HyukjinKwon
  and
  Isaac
  committed
  a6a287c
- Add behavioral functions e2e (math/string/conditional/collection/aggregate)
  
  HyukjinKwon
  and
  Isaac
  committed
  1e6ac8d
- Cover catalog create_table/refresh/recover_partitions (client-side DDL path)
  
  HyukjinKwon
  and
  Isaac
  committed
  9a34c8a
- Exercise the UDF-builder client paths (group/dataframe applyInPandas/mapInArrow/...)
  
  HyukjinKwon
  and
  Isaac
  committed
  462912c
- Extend functions golden 440 -> 508 (byte-for-byte vs reference)
  
  HyukjinKwon
  and
  Isaac
  committed
  f6aacd2
- Address re-review: V2 writer option coercion, run e2e in CI, plan-serialization guard
  
  HyukjinKwon
  and
  Isaac
  committed
  814cdbd
- rustfmt: apply formatting to e2e_columns + plan_arg_regression_golden
  
  HyukjinKwon
  and
  Isaac
  committed
  763275e
- functions: match reference builder args for uuid + tuple_sketch_agg_{double,integer}
  
  HyukjinKwon
  and
  Isaac
  committed
  b3ab882
- catalog: return catalog list ops as lazy relations (fix empty-result error)
  
  HyukjinKwon
  and
  Isaac
  committed
  3559267
- table_arg: implement TableArg::to_proto + replace empty stub test
  
  HyukjinKwon
  and
  Isaac
  committed
  8e37d73
- session: support NULL values in createDataFrame typed columns (+ cover all arms)
  
  HyukjinKwon
  and
  Isaac
  committed
  fbfb379
- window: unit-test WindowSpec/Window builders (was ~52% covered)
  
  HyukjinKwon
  and
  Isaac
  committed
  cbc9d8c
- row: cover Value Display + accessors + Row helpers (was ~67%)
  
  HyukjinKwon
  and
  Isaac
  committed
  233e6d5
- tvf: cover variant_explode / variant_explode_outer / python_worker_logs
  
  HyukjinKwon
  and
  Isaac
  committed
  c7ae47a
- catalog: cover the db/pattern-qualified + metadata API variants (100% API)
  
  HyukjinKwon
  and
  Isaac
  committed
  a730b58
- readwriter: cover saveAsTable/insertInto/bucketBy/sortBy + writerV2 (API)
  
  HyukjinKwon
  and
  Isaac
  committed
  bd83612
- streaming: cover file-source readers + name + process_all_available
  
  HyukjinKwon
  and
  Isaac
  committed
  0e72ba4
- dataframe: implement `to` (schema reconcile) + `with_metadata` (were stubs)
  
  HyukjinKwon
  and
  Isaac
  committed
  7dd2faf
- ml: implement Evaluator.evaluate via MlCommand (was a stub) + cover ML client paths
  
  HyukjinKwon
  and
  Isaac
  committed
  f43e61c
- readwriter: add xml reader/writer + expose jdbc/xml in the Python drop-in
  
  HyukjinKwon
  and
  Isaac
  committed
  d3d4366
- Fix 3 confirmed stubs found by the stub sweep: transpose, zipWithIndex, explain
  
  HyukjinKwon
  and
  Isaac
  committed
  0c5b87a
- column: expose name/astype/try_cast/getItem + cast accepts DataType or str (drop-in)
  
  HyukjinKwon
  and
  Isaac
  committed
  fddaf6b
- functions: add the optional-arg variants (round/ceil/floor/bround scale, trim char, to_binary format, approx_count_distinct rsd)
  
  HyukjinKwon
  and
  Isaac
  committed
  df01b03
- drop-in parity: sample/unionByName/tail/DecimalType/Window/Builder signatures
  
  HyukjinKwon
  and
  Isaac
  committed
  2370bf1
- types: make the simple DataType classes instantiable with simpleString/typeName
  
  HyukjinKwon
  and
  Isaac
  committed
  e7465b4
- types: real DataType object model (StructType/StructField/Array/Map) + converter
  
  HyukjinKwon
  and
  Isaac
  committed
  d8125ee
- dataframe drop-in: expose transpose / lateralJoin / repartitionByRange
  
  HyukjinKwon
  and
  Isaac
  committed
  618a131
- Fix CI (drop removed zip_with_index test) + add the 7 complex DataType classes
  
  HyukjinKwon
  and
  Isaac
  committed
  29fc618
- column: alias(name, metadata=...) attaches column metadata
  
  HyukjinKwon
  and
  Isaac
  committed
  cea2850
- readwriter: reader format methods accept per-call **options
  
  HyukjinKwon
  and
  Isaac
  committed
  77e95b1
- dataframe: expose df-level approxQuantile / crosstab / freqItems
  
  HyukjinKwon
  and
  Isaac
  committed
  fbaf87f
- readwriter: full jdbc signature (column/bounds/numPartitions/properties)
  
  HyukjinKwon
  and
  Isaac
  committed
  e288a90
- types: DecimalType exposes precision/scale attrs + simpleString/typeName/repr
  
  HyukjinKwon
  and
  Isaac
  committed
  6a51ac8
- ml: expose pyspark.ml.connect Python bindings (the 11 core ML classes)
  
  HyukjinKwon
  and
  Isaac
  committed
  08cc3d2
- dataframe: expose groupingSets/toJSON/dropDuplicatesWithinWatermark + server-backed sameSemantics/semanticHash
  
  HyukjinKwon
  and
  Isaac
  committed
  1d7ced9
- Fix e2e_dataframe for the Result-returning same_semantics/semantic_hash
  
  HyukjinKwon
  and
  Isaac
  committed
  2602f6b
- readwriter: full explicit pyspark signatures for reader csv/json/orc/text/xml
  
  HyukjinKwon
  and
  Isaac
  committed
  36e118a
- dataframe: full explicit pyspark signatures for writer csv/json/parquet/orc/text/xml
  
  HyukjinKwon
  and
  Isaac
  committed
  2c0ea50
- Expose SparkSession.tvf, copyFromLocalToFs, Observation, and DataFrame.observe
  
  HyukjinKwon
  and
  Isaac
  committed
  072aa08
- ml: add MulticlassClassificationEvaluator + CrossValidator(+Model) (core + bindings)
  
  HyukjinKwon
  and
  Isaac
  committed
  f24e7d9
- drop-in: writer.partitionBy accepts nested list + createDataFrame AtomicType schema
  
  HyukjinKwon
  and
  Isaac
  committed
  7a9c6d1
- parity gate: retry test_parity_arrow_python_udf (transient worker-spawn timeout)
  
  HyukjinKwon
  and
  Isaac
  committed
  0a7e330
- ml: unit-test Multiclass/CrossValidator + every estimator's fit/transform/clone_box
  
  HyukjinKwon
  and
  Isaac
  committed
  a36703a
- udf: make the drop-in udf() path actually work end-to-end
  
  HyukjinKwon
  and
  Isaac
  committed
  dafe645
- Close all public-API parity gaps + WASM UDF SQL registration
  
  HyukjinKwon
  and
  Isaac
  committed
  99cd5d2
- Add Rust + Python coverage badges and a coverage CI workflow
  
  HyukjinKwon
  and
  Isaac
  committed
  eb96a5e
- docs: finalize README with coverage badges and API parity statement
  
  HyukjinKwon
  and
  Isaac
  committed
  e3d54df
- README: real repo slug for coverage badges, parity + WASM UDF sections
  
  HyukjinKwon
  and
  Isaac
  committed
  d918991
- Add comprehensive offline unit tests for core Rust crate (78 tests)
  
  HyukjinKwon
  and
  Isaac
  committed
  627a75b
- Add Python drop-in offline tests, coverage scoping, and server-gated e2e
  
  HyukjinKwon
  and
  Isaac
  committed
  84c629c
- Re-vendor cloudpickle 3.1.1 -> 3.1.2 to match Spark v4.2.0
  
  HyukjinKwon
  and
  Isaac
  committed
  98a3cf1
- WASM UDF spark.udf().register(); fix metadata JSON re-quoting; coverage + workflow
  
  HyukjinKwon
  and
  Isaac
  committed
  11dafff
- Fix clippy errors + quiet warnings; add vendored-upstream drift CI
  
  HyukjinKwon
  and
  Isaac
  committed
  47dca74
- AGENTS.md: full version-upgrade parity playbook
  
  HyukjinKwon
  and
  Isaac
  committed
  798ff55
- Raise core coverage past 90% (region/function); bound the parity CI job
  
  HyukjinKwon
  and
  Isaac
  committed
  daf6229
- Re-vendor serializers.py + add version.py from Spark v4.2.0
  
  HyukjinKwon
  and
  Isaac
  committed
  ad10539
- Parity gate: give slow worker-heavy files a larger per-file timeout
  
  HyukjinKwon
  and
  Isaac
  committed
  47be40d
- Embed the WASM UDF packer in the crate; drop the pyspark_wasm_udf package
  
  HyukjinKwon
  and
  Isaac
  committed
  085d95d
- Embed the WASM UDF packer script + wire it up (follow-up to package removal)
  
  HyukjinKwon
  and
  Isaac
  committed
  bcac60c
- AGENTS.md: always merge PRs with the dev merge script
  
  HyukjinKwon
  and
  Isaac
  committed
  f9eb3d7

Commits

Commits on Aug 25, 2026

Commits on Aug 26, 2026