chore: sync spark4.1 with master

.github/skills/scala-code/SKILL.md

@@ -0,0 +1,49 @@
+---
+name: scala-code
+description: Write and modify Scala code in SynapseML. Use when adding or changing Scala transformers, estimators, or params to follow repo patterns, keep scalastyle and compilation green, and cover behavior changes with tests.
+---
+
+# Scala Code Skill
+
+Use this skill for any Scala code change in this repository.
+
+## Objectives
+- Keep Scala changes correct, minimal, and production-safe.
+- Prevent CI/CD breakage by validating style and compilation before completion.
+- Ensure every behavior change is covered by tests.
+- Scala code should be optimized to be efficient and maintainable, following existing patterns and practices in the codebase.
+
+## Scala best practices
+- Follow existing SynapseML patterns (`DefaultParamsReadable`, `DefaultParamsWritable`, `Wrappable`, `SynapseMLLogging`).
+- Keep business logic in Scala (not generated Python wrappers).
+- Do not edit generated files under `target/`.
+- Preserve license headers and existing package structure.
+- Prefer small, focused changes and reuse existing helpers/traits.
+- Avoid introducing flaky tests or network-dependent behavior unless already required by the suite.
+- To get around scalastyle issues, don't just use `// scalastyle:off` or `//scalastyle:ignore`, but instead fix the underlying issue or refactor to avoid it. Unless, the refactor results in a more complex code structure, in which case, it may be acceptable to disable the specific scalastyle rule for that line or block of code, but this should be done sparingly and with justification.
+
+## Required validation steps (must run)
+Run these commands before finishing Scala changes:
+
+1. Scala style checks:
+   - `sbt scalastyle "Test / scalastyle"`
+2. Scala compile checks:
+   - `sbt compile`
+   - `sbt test:compile`
+3. Relevant tests for touched modules/files:
+   - Example: `sbt "core/testOnly *SuiteName*"` or `sbt "cognitive/testOnly *SuiteName*"`
+
+If a command fails, fix the issue and rerun until passing.
+
+## Testing requirement for code changes
+- Any Scala code change must be accompanied by tests (new tests or updates to existing tests).
+- Bug fixes must include a regression test that fails before the fix and passes after.
+- New logic/branches should include coverage for success and failure/edge cases where practical.
+- Keep tests deterministic and aligned with current module test conventions.
+
+## Completion checklist
+- [ ] Code follows existing Scala/SynapseML conventions.
+- [ ] Style checks pass.
+- [ ] Compile checks pass.
+- [ ] Relevant tests pass.
+- [ ] Scala code changes include corresponding tests.

.github/skills/synapseml-branches/SKILL.md

@@ -0,0 +1,57 @@
+---
+name: synapseml-branches
+description: >-
+  Resolve SynapseML branch-specific rules, runtime baselines, sync policy, and
+  CI expectations. Use before editing, rebasing, merging, testing, or declaring
+  readiness for master, spark3.5, spark4.0, spark4.1, port/sync branches, or a PR
+  whose base branch determines behavior.
+compatibility: SynapseML repository with git and GitHub CLI.
+---
+
+# SynapseML branch context
+
+Use the PR base branch as the context. A feature branch name does not determine
+the runtime, sync policy, or CI that must pass.
+
+## Workflow
+
+1. Read root `AGENTS.md`.
+2. Resolve the target:
+   - For a PR, read `baseRefName` from GitHub.
+   - For direct branch work, use the checked-out shared branch.
+3. Load the mapped reference. Filenames use `p` for the version decimal:
+   - `master` ->
+     [branch-spark3p5.md](references/branch-spark3p5.md). `master` is currently
+     the canonical Spark 3.5 development baseline.
+   - `spark3.5` ->
+     [branch-spark3p5.md](references/branch-spark3p5.md). This is the shared
+     Spark 3.5 release branch.
+   - `spark4.0` ->
+     [branch-spark4p0.md](references/branch-spark4p0.md).
+   - `spark4.1` ->
+     [branch-spark4p1.md](references/branch-spark4p1.md).
+   - Any other target ->
+     [branch-fallback.md](references/branch-fallback.md).
+4. Verify every version, dependency, trigger, skip, and test command against
+   that branch's live `build.sbt`, `environment.yml`, workflows, and
+   `pipeline.yaml`. References are decision guides, not stale-value authority.
+5. Recheck branch context at three points: before implementation, before
+   validation, and immediately before push/readiness. Target movement or a
+   changed base invalidates earlier evidence.
+
+## Responsibilities
+
+- Ordinary PRs rebase onto their latest target with `--force-with-lease`.
+- Shared port branches receive `master` by merge; never rebase or force-push
+  the shared branch.
+- For conflict resolution, compare merge base, `master`, and port branch;
+  ancestry alone does not prove both sides survived.
+- Inspect CI definitions on the target branch and confirm builds actually
+  queued. Never infer port-branch coverage from `master`.
+- Confirm the relevant suites ran by test result/class, not only job status.
+- Treat `.github/skills/` as authoritative. `.agents/` is compatibility-only
+  and may contain stale copies.
+- If no exact reference exists, follow the fallback, state uncertainty, and
+  add a concise reference from the
+  [branch template](references/branch-template.md) when the branch is an active
+  supported target.

.github/skills/synapseml-branches/references/branch-fallback.md

@@ -0,0 +1,14 @@
+# Unlisted target branch
+
+- Determine whether the target is an ordinary feature branch, a shared port
+  branch, a release branch, or an automation branch.
+- Read its live build files, workflows, pipeline triggers, recent sync PRs, and
+  open/closed PRs targeting it.
+- Do not guess versions, supported environments, merge policy, or CI coverage
+  from the branch name.
+- Rebase ordinary PRs; merge into shared branches unless repository history
+  proves a different maintained policy.
+- Confirm actual queued checks and per-test execution.
+- State unresolved branch-specific uncertainty as a blocker. If the target is
+  actively supported, add a focused reference using
+  [branch-template.md](branch-template.md) before claiming readiness.