Repository navigation
[SPARK-59033][INFRA][DOCS] Coverage badges at branch root + fix udfs page rendering - #59
Closed
HyukjinKwon wants to merge 6 commits into
Closed
HyukjinKwon wants to merge 6 commits into
HyukjinKwon wants to merge 6 commits into
Conversation
…branch root
The README shields.io endpoints read .../badges/coverage-rust.json (root of the
badges branch), but coverage.yml wrote the JSON into a badges/ subdirectory, so the
published path was .../badges/badges/coverage-rust.json and the badges 404'd
('resource not found'). Write coverage-rust.json / coverage-python.json at the
branch root instead.
Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
changed the title
The udfs docs page mangled type names (Vec<T>, Option<T>, Vec<Option<String>>, udf::<name>): two inline code spans wrapped across a line, so with md_in_html the bare <...> after the unclosed span were parsed as HTML tags and dropped. Collapse those spans onto one line. Also drop the obsolete 'export SPARK_CONNECT_WASM_PACKER_PATH=...' line from the run example — the packer is now embedded in the crate, so no path is needed. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
changed the title
The four ```rust,ignore fences on the UDFs page never rendered as code: pymdownx.superfences rejects a comma in the fence info string, so the fence isn't recognized at all - the backticks show up literally and the `#[spark_wasm_udf]` / `#[cfg(...)]` lines inside get parsed as H1 headings (see the reported screenshot). `rust,ignore` is rustdoc/mdBook doctest syntax, not mkdocs; the site just wants a language, so use ```rust. Verified by rendering the whole page through the site's exact extension set: 7 code blocks, no stray backticks, no bogus headings. Co-authored-by: Isaac <no-reply@databricks.com>
The README's UDF section led with the low-level factory (read a prebuilt
`.wasm`, spell out AbiTypes by hand), which isn't how UDFs are written in
other languages. Lead instead with the self-contained macro flow: define
plain Rust functions with `#[spark_wasm_udf]` and call them directly
(`udf::add_one(col("id"))`) or register by name for SQL
(`spark.udf().register("shout", &udf::shout_udf())`), all in one file --
matching the `examples/wasm-udf-inline/` example. The lower-level factory
is now a pointer into the guide.
Verified against the crate: range(5)? -> Result, select -> DataFrame,
Column::alias consumes self, SparkSession::udf()/register take
&UserDefinedFunction, and the macro emits udf::<name>_udf() ->
UserDefinedFunction.
Co-authored-by: Isaac <no-reply@databricks.com>
…ction) Drop the standalone "API Parity & Drop-in Replacement" section and fold its claim into a single clause in the intro. Shorten the UDF section to a minimal define-and-call example and defer the full setup / SQL registration / supported types to the WASM UDF guide. Co-authored-by: Isaac <no-reply@databricks.com>
viirya
approved these changes
Aug 26, 2026
…e check
My previous fix flipped the UDFs page from ```rust,ignore to ```rust so
superfences would render it. That also opted those blocks INTO
check_doc_snippets.py (which compiles every plain ```rust block), and
they can't compile in that harness -- they use the `wasm-udf`
`#[spark_wasm_udf]` macro, a build.rs embedding step, and the generated
`udf::*` module -- so the Docs job's snippet-compile step failed.
Use the ```{.rust .no-run} brace fence instead: superfences still renders
it as a Rust code block (verified through the site's exact extension
set), but the checker's `^```rust$` regex doesn't match it, so it's
skipped -- the renderable equivalent of rustdoc's `ignore`. Document the
convention in check_doc_snippets.py so `rust,ignore` (which breaks
rendering) isn't reintroduced.
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Publish the coverage badge JSON at the root of the
badgesbranch(
coverage-rust.json,coverage-python.json) instead of inside abadges/subdirectory.
Why are the changes needed?
The README shields.io endpoint badges read:
https://raw.githubusercontent.com/apache/spark-connect-rust/badges/coverage-rust.jsonhttps://raw.githubusercontent.com/apache/spark-connect-rust/badges/coverage-python.jsoni.e. files at the root of the
badgesbranch. Butcoverage.ymlcopied them into abadges/subdirectory before committing, so the published path was.../badges/badges/coverage-rust.jsonand the badges rendered as "resource notfound" (404). This writes the JSON at the branch root so the endpoints resolve. It
takes effect on the next master push (the workflow publishes on push to master).
Does this PR introduce any user-facing change?
No — CI/badge publishing only.
How was this patch tested?
YAML validated; the publish step now
cps the badge JSON to the branch root andgit adds the root files. Confirmed the README URLs point at the root paths.This pull request and its description were written by Isaac.