feat: Automate release preparation, notes, and artifact verification

[Rana Singh (ranadeepsingh)]

What

Automates the mechanical parts of the SynapseML Fabric release, and fixes scripts/bump-version.py, which currently fails on master and would block the next release.

Complements #2540 (which automates tagging and the spark rebase PRs). No overlap: this PR covers the version bump, the GitHub Release, and the artifact/BBC-VHD side.

Why

scripts/bump-version.py hard-fails on current master. #2589 (CDN migration) rewrote docs/Reference/R Setup.md to per-module archive names (synapseml-core-1.1.3.zip) and added VerifyRCodegen.scala with hardcoded version strings. Neither is covered by the existing anchors, so the script exits non-zero before doing anything useful.

Beyond that, three things about the current process show up clearly in the history:

1. Version bumps bypass review. v1.1.0 and v1.1.3 landed as unsigned direct pushes to master with no PR. v1.1.2 was bumped (baf63dce) and then removed by resetting master back to its parent, with no revert PR — the tag simply never existed.

2. GitHub Releases are inconsistent. v1.1.1 has a tag but no Release object, and neither do any of the v1.1.3-spark4.* / -python3.* tags. Because GitHub anchors auto-generated notes to the previous Release rather than the previous tag, v1.1.3's notes silently spanned both v1.1.1 and v1.1.3.

3. Version strings are hand-typed across four inconsistent conventions. The same release is spelled four different ways, and the differences are invisible unless two real releases are compared side by side:

   OSS UPack       1.1.3-spark4-0       spark dots become dashes
   Internal UPack  1.1.3-0-spark4.0     spark dots are preserved
   OSS pip         1.1.3+python3.12     PEP 440 local segment
   Internal pip    1.1.3.0+python3.12   super-patch, then local segment
   

Changes

scripts/bump-version.py (fixes)

Bug	Effect
docs/Reference/R Setup.md + VerifyRCodegen.scala not matched after #2589	script hard-fails on master
_detect_version() read docusaurus.config.js with no encoding	UnicodeDecodeError on a cp1252 console
analyze() and the EXPECTED_FILES check compared str(rel)	backslashes never match the / anchor tables on Windows; false "not updated" warnings
non-ASCII status output on a cp1252 console	crashed after rewriting files, leaving a half-applied bump behind a non-zero exit

Also adds DENYLIST_PATHS for files whose basename is too common to denylist safely.

scripts/release/ (new)

• release_matrix.py — derives every tag, UPack version, pip version and BBC-VHD value from one input version, so the four conventions above are applied by code rather than by hand. --json for pipeline consumption.
• verify_release.py — checks every tag and published artifact against the matrix. Worth running even on a green publish pipeline, because several publish steps in SynapseML-Publish-Official use continueOnError: true and can fail silently.
• bump_bbcvhd.py — applies a release to a BBC-VHD component (setup.sh + version.txt), replacing the most error-prone hand-edit in the process.

.github/workflows/ (new)

• release-prepare.yml — workflow_dispatch(version) runs the bump and opens the release PR, so a release can't be an unreviewed direct push. Refuses to run off master, or if the tag or branch already exists.
• release-notes.yml — publishes the GitHub Release when a vX.Y.Z tag is pushed, pinning the diff base to the previous primary tag so notes stay correct even when an older Release object is missing. Verifies the tag is an ancestor of master first.

Validation

• 192 passed (test_bump_version.py + test_release_matrix.py). On this branch's base commit the same suite is 7 failed / 166 passed / 4 errors on Windows.
• verify_release.py --version 1.1.3 → 26 checks, 0 missing, exit 0 against the live feeds.
• verify_release.py --version 1.1.1 → 5 missing, exit 1, correctly detecting a real historical gap.
• release_matrix.py reproduces the live BBC-VHD dev/spark40/setup.sh byte-for-byte, including the -1 rebuild counter on the OSS package and its absence on the Internal one.
• bump_bbcvhd.py run against a copy of the real BBC-VHD files produces a 2-line diff with no line-ending damage.
• actionlint clean on both workflows.
• black==22.3.0 (the pinned CI version) clean on all changed Python.

Expected values in the tests are transcribed from live v1.1.1 / v1.1.3 data rather than from documentation, so a failure means the tooling has drifted from what was actually shipped.

Note for #2540

release-tag.yml creates v<version>-python3.11 on master but not v<version>-spark3.5. That tag exists for both released versions that use the convention:

v1.1.3 -> a833941704 : v1.1.3, v1.1.3-python3.11, v1.1.3-spark3.5
v1.1.1 -> 0f2b899b51 : v1.1.1, v1.1.1-python3.11, v1.1.1-spark3.5

Worth adding there, since it's the tag the Spark 3.5 publish leg keys off.

Not included

Anything that crosses into Azure DevOps (the SynapseML-Internal bump, triggering the publish pipeline, opening the BBC-VHD PR) is deliberately left out of this repo — it would need an ADO PAT in a public repository. That half belongs in the ADO SynapseML-OSS repo, which already has a GitHub service connection and can drive both sides.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.