Skip to content

docs: add concise agent and PR readiness guidance - #2649

Merged
Rana Singh (ranadeepsingh) merged 13 commits into
masterfrom
docs/agents-branch-model
Aug 17, 2026
Merged

docs: add concise agent and PR readiness guidance#2649
Rana Singh (ranadeepsingh) merged 13 commits into
masterfrom
docs/agents-branch-model

Conversation

@ranadeepsingh

@ranadeepsingh Rana Singh (ranadeepsingh) commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace the long root guide with a concise decision-oriented AGENTS.md
  • add synapseml-pr-loop for repeatable 5/5 merge-readiness work
  • add synapseml-branches with focused context for master, spark3.5, spark4.0, spark4.1, and unlisted targets
  • keep Spark 4 context concise through a shared template and branch overlays templatized from chore: sync spark4.1 with master #2645 and chore: sync spark4.0 with master #2646
  • map master and spark3.5 to branch-spark3p5.md, with separate sync rules
  • require lifecycle cleanup for recently closed work, follow-up rebases/closures, and current human-readable PR metadata
  • encode historical false-confidence cases: stale targets, blocking votes, suppressed comments, unselected suites, path-filter bypasses, unshipped artifacts, and target-branch CI gaps

Why

Recent SynapseML PRs showed that green aggregate CI is insufficient. Agents must resolve context from the PR base branch, verify the target branch's live build and CI definitions, prove the public shipped path, inspect every review surface, and recheck context before implementation, validation, and final push.

Detailed checklists remain in focused references so the always-loaded guidance stays terse.

Validation

  • all relative links resolve
  • skill names match their directories and follow the Agent Skills specification
  • synapseml-pr-loop is 139 lines; synapseml-branches is 57 lines
  • no skill line exceeds 120 characters
  • readiness snapshot script executed against live PRs
  • branch is based on current master
  • git diff --check passes

Historical inputs include #2608, #2611, #2617, #2622, #2627, #2631-#2638, and #2644.

Follow-up to #2648.

Copilot AI lite review requested due to automatic review settings August 17, 2026 01:58
@github-actions

Copy link
Copy Markdown

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

  • fix: Fix LightGBM crashes with empty partitions
  • feat: Make HTTP on Spark back-offs configurable
  • docs: Update Spark Serving usage
  • build: Add codecov support
  • perf: improve LightGBM memory usage
  • refactor: make python code generation rely on classes
  • style: Remove nulls from CNTKModel
  • test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

## Summary`nReduce AGENTS.md from a long duplicated reference to a short repository decision guide with direct links to authoritative build, setup, review, codegen, testing, branch, and CI sources.

## Prompting Intent`nEnsure every instruction in the new AGENTS.md is helpful and terse, and replace copied detail with links and pointers to what coding agents actually need.

## Linked Sources`n- Pull request: https://github.com/microsoft/SynapseML/pull/2648`n- Contributor guide: CONTRIBUTING.md`n- Repository skills: .github/skills/`n- Build sources: build.sbt, environment.yml, pyproject.toml, pipeline.yaml

## Rationale`nAgents need high-signal boundaries and navigation, not a second copy of implementation examples. Keeping durable rules while linking to source files reduces staleness, token cost, and branch-sync conflicts without losing actionable guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors AGENTS.md into a shorter, decision-focused agent guide, keeping durable repo-wide rules while pointing readers to authoritative sources for versions, setup, codegen, testing, branch practices, and CI.

Changes:

  • Condenses AGENTS.md substantially while preserving key non-negotiable rules and branch-model guidance.
  • Replaces version-/example-heavy prose with links to source-of-truth files and repo Skills.
  • Keeps a compact repository map and a checklist for Scala-first/codegen conventions and validation steps.
Show a summary per file
File Description
AGENTS.md Rewrites the agent guide to be concise and link-driven while retaining core rules, branch model, and validation guidance.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

AGENTS.md and CONTRIBUTING.md are required to be identical on every branch,
but both enumerated the port branches by name -- three places in AGENTS.md
and one in CONTRIBUTING.md. That makes adding a port branch an edit to a
file that must then be re-synced everywhere, which is the exact churn the
identical-everywhere rule exists to avoid.

Describe the pattern instead of listing instances: port branches are named
spark<version>, and `git branch -r` is the authoritative list. Same reason
version numbers are read from build.sbt rather than restated -- an
enumeration in prose is a copy that goes stale silently.

State the boundary explicitly in "Keep this file useful", since it was
implied rather than written: these two files may not name a Spark, Scala,
Java or Python version, or a path containing one, while README, the website
and module docs are free to be branch- and version-specific because nothing
requires those to match across branches.

Verified: no version-like token remains in either file, and every relative
link still resolves.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 02:25
## Summary`nAdd a project skill for taking SynapseML issues and stale PRs through an evidence-based merge-readiness loop, with reusable readiness gates, CI triage, Spark performance guidance, and a live GitHub snapshot script. Link it tersely from AGENTS.md.

## Prompting Intent`nCapture the recurring 5/5 or 200%-ready development workflow from prior SynapseML sessions so agents consistently rebase, resolve all active and suppressed feedback, prove user value, add regression and end-to-end tests, protect compatibility and Spark performance, and iterate full CI to green.

## Linked Sources`n- Follow-up PR: https://github.com/microsoft/SynapseML/pull/2649`n- Original agent guide PR: https://github.com/microsoft/SynapseML/pull/2648`n- Project review skill: .github/skills/code-review/SKILL.md`n- Project local setup skill: .github/skills/synapseml-local-setup/SKILL.md`n- Agent Skills specification: https://agentskills.io/specification

## Rationale`nA dedicated skill provides repeatable progressive disclosure without bloating AGENTS.md. The workflow encodes evidence gates learned from real failures: stale targets, discarded conflict content, suppressed comments, helper-only tests, false-green skips, infrastructure failures, compatibility breaks, unshipped artifacts, and unmeasured Spark performance claims.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread AGENTS.md Outdated
Copilot AI review requested due to automatic review settings August 17, 2026 02:28
…idence

Two statements in the new skill would send an agent down a path I measured
as wrong while getting the spark4.0 and spark4.1 sync PRs green.

"Push the exact validated head and comment /azp run" does not work for PRs
targeting the port branches. The Azure Pipelines definition's pull-request
trigger is defined in the pipeline UI with a branch filter of +master, and a
UI-defined trigger overrides the pr: block in pipeline.yaml entirely, so the
YAML listing the port branches has no effect. /azp run on such a PR silently
does nothing -- no build queues and no error is reported -- which reads as
"CI triggered" and then as "CI pending" forever. Replace it with: trigger,
then confirm a build actually queued, and queue against refs/pull/N/merge
when the target is not covered. Cite the build ID, since a comment is not
evidence a build ran.

CI triage described the four failure categories but not how to read a job
result, and the mechanics are not binary. A filter of result -eq "succeeded"
reports phantom failures, because succeededWithIssues is a normal outcome
when a non-gating task -- usually dependency caching or TLS -- warns while
every test passes. I hit exactly this and briefly reported a passing job as
failed. It also cuts the other way: succeededWithIssues on a task that runs
or publishes tests is a real failure. Add a section saying to identify the
warning task and read published test results rather than trusting the badge
in either direction.

Same section records the harder lesson: compare per-test outcomes across
builds. A fix that changes nothing leaves the same tests failing the same
way, and a job-level summary hides that. Two changes I believed were fixes
turned out to be placebos under that comparison.

Also drop the remaining version numbers, matching the previous commit --
"spark4.x" becomes "spark<version>" and "Spark 4.1 compatibility" becomes
"port-branch compatibility", so adding a branch does not require editing
these files.

Verified: no version token remains in the skill, and both sibling skill
links resolve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 4
  • Review effort level: Lite

Comment thread .github/skills/synapseml-merge-ready/scripts/Get-PrReadiness.ps1 Outdated
Comment thread .github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1
Comment thread .github/skills/synapseml-merge-ready/scripts/Get-PrReadiness.ps1 Outdated
Comment thread .github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1
Copilot AI review requested due to automatic review settings August 17, 2026 02:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

.github/skills/synapseml-merge-ready/scripts/Get-PrReadiness.ps1:77

  • gh api compare endpoint is built with $view.baseRefName unescaped. If the base branch contains / (common for release/hotfix branches), this produces an invalid REST path and the script will fail even though the PR data is otherwise valid. URL-encode the base ref before constructing the compare URL.
    $compareText = & gh api "repos/$Repo/compare/$($view.baseRefName)...$($view.headRefOid)"

.github/skills/synapseml-merge-ready/scripts/Get-PrReadiness.ps1:97

  • Detection of suppressed Copilot review bodies is case-sensitive and requires the exact substring "Suppressed comments". GitHub review templates/messages can vary in casing, which will cause false negatives and silently drop the signal this script is meant to surface. Use a case-insensitive check.
    $suppressed = @($review.reviews.nodes | Where-Object {
        $_.body -and $_.body.Contains("Suppressed comments")
    } | ForEach-Object {

.github/skills/synapseml-merge-ready/scripts/Get-PrReadiness.ps1:7

  • The help text claims the script reports "active review threads" and "review bodies containing suppressed comments" without noting that the GraphQL query is capped (reviewThreads(first: 100), comments(first: 20), reviews(last: 50)). For large PRs this can produce incomplete evidence without warning; the help should document these limits (or the script should paginate).
.DESCRIPTION
    Reports head/base state, target divergence, checks, active review threads,
    and review bodies containing suppressed comments. It does not make the
    readiness decision; use the skill's evidence gates for that judgment.
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Review feedback on the script, verified case by case rather than applied
wholesale. A readiness tool that under-reports is worse than no tool, so the
truncation issues were the ones worth fixing properly.

Pagination. The GraphQL query capped reviewThreads at 100, per-thread comments
at 20 and reviews at 50, with no pageInfo, so a large PR would silently report
fewer unresolved threads than it has -- reading as "review is clean" when it
is not. That is the false-green pattern the skill itself tells you to reject.
Both connections are now fully paginated through a shared helper, and the
emitted JSON carries a completeness object with page counts and any thread
whose comments were still truncated, so an incomplete snapshot is visible
instead of silent. The loop throws if a page claims hasNextPage without
returning a cursor rather than spinning forever.

Suppressed-comment detection was an exact case-sensitive match on
"Suppressed comments", which silently drops the signal the script exists to
surface if GitHub varies the wording. Now a case-insensitive match.

Repo validation accepted "owner/name/extra", because Split("/", 2) always
yields two parts when a slash is present. Now requires exactly two non-empty
segments and reports the offending value.

Renamed the helper's local $args to $ghArgs; $args is an automatic variable
inside a function and assigning to it is a trap for later edits.

Two review points I did not treat as defects:

statusCheckRollup was reported as an object whose checks live under .contexts.
That is the GraphQL shape, but `gh pr view --json` flattens it. Measured: it
returns Object[] of 13 CheckRun entries with name/status/conclusion. Verified
the existing filter against a PR that genuinely fails and it returned exactly
[Review Dependencies], matching `gh pr checks`. Left as is -- an all-green PR
would not have proven this either way.

The unescaped base ref in the compare URL was reported as breaking on branches
containing "/". It does not: the API accepts sync/spark4.1-with-master-2
unescaped and returns the same result as the encoded form. Kept the encoding
anyway as defensive, but it fixes no observed failure.

Verified: parses clean, and runs against three PRs producing counts that match
independent REST pagination.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 02:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/skills/synapseml-merge-ready/references/ci-triage.md:28

  • Grammar: this reads as though the issue itself has an owner. Consider using “owning issue” to mean the issue that tracks the baseline failure.
Action: collect comparable target/head evidence. Do not silently ignore it; link
the owner issue or repair it when tightly coupled.
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

## Summary
Rename the reusable skill and its directory from synapseml-merge-ready to synapseml-pr-loop, then update the AGENTS.md activation link.

## Prompting Intent
The engineer asked for a clearer, more understandable skill name that describes the recurring SynapseML pull-request remediation loop.

## Linked Sources
- Pull request: #2649
- Follow-up context: #2648

## Rationale
SynapseML PR loop communicates the skill's repeatable issue/PR workflow more directly than the outcome-oriented merge-ready name while preserving all existing progressive-disclosure content and behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 02:54
## Summary
Add the remaining concise instructions needed for repeatable PR remediation: audit historical review feedback, verify published artifacts ship the capability, update public documentation without editing generated files, and safely validate external services.

## Prompting Intent
The engineer asked for the SynapseML PR loop to contain all recurring instructions and checklists so future merge-readiness requests do not require repeated guidance, while keeping the skill terse.

## Linked Sources
- Pull request: #2649
- Original guide: #2648

## Rationale
Keep the main workflow at 114 lines and place detailed exit, CI, and Spark checks in focused references. The added bullets close material workflow gaps without duplicating those references.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment thread .github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1
Comment thread .github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1
Comment thread .github/skills/synapseml-pr-loop/SKILL.md
Copilot AI review requested due to automatic review settings August 17, 2026 02:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1:109

  • The loop guard comment says it protects against a cursor that "never advances", but the current check only handles a missing cursor. If the GraphQL API ever returns the same endCursor with hasNextPage=true, this can still loop forever. Capture the previous cursor and also throw when the cursor does not advance.
        # Guard against a cursor that never advances rather than looping forever.
        if ($hasNext -and -not $cursor) {
            throw "$Description reported more pages but returned no cursor for PR #$Number"
        }
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

## Summary
Add a concise synapseml-branch skill with focused master, spark3.5, spark4.0, spark4.1, and fallback references. Wire AGENTS.md and the PR loop to resolve context from the PR base branch and add historical false-confidence gates.

## Prompting Intent
The engineer asked to mine prior SynapseML PRs and Copilot sessions for durable lessons, make agent responsibilities and repeated checks explicit, provide branch-specific shared context, and ensure agents fall back to it without bloating the main PR loop.

## Linked Sources
- Pull request: #2649
- Branch CI coverage: #2644
- Spark 4.1 synchronization: #2617
- Compatibility replay fix: #2611
- Compatibility identity fix: #2608
- Orphaned test-suite coverage: #2622

## Rationale
Use .github/skills because branch-local .agents guidance identifies it as authoritative. Keep common decision logic in one small skill, isolate volatile branch facts in references, derive context from the PR base rather than feature-branch names, and require live build/CI verification so historical notes cannot become stale authority.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 03:07
## Summary
Handle GraphQL errors and missing data explicitly, reject non-advancing cursors, classify stale checks as failures, emit a stable JSON array shape, and warn that review content must remain local or be redacted.

## Prompting Intent
The engineer asked agents to double- and triple-check the reusable PR loop. The loop's own final snapshot surfaced active and suppressed Copilot findings that needed to be fixed before the skill could be considered reliable.

## Linked Sources
- Pull request: #2649
- GraphQL error review: #2649 (comment)
- Stale-check review: #2649 (comment)
- Snapshot privacy review: #2649 (comment)

## Rationale
A readiness collector must fail closed and preserve a stable machine-readable contract. Explicit errors, cursor progress checks, stale-signal blocking, array output, and local/redacted evidence prevent false-green or accidental disclosure outcomes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 17, 2026 03:09
@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

Final review follow-up on a6eb3fdca7:

  • GraphQL errors/missing data now fail explicitly.
  • Pagination rejects missing or repeated cursors.
  • STALE checks block readiness.
  • Single- and multi-PR output always use the same JSON array shape.
  • Snapshot content is documented as local/sensitive and must be redacted before sharing.
  • The suppressed wording issue now says “tracking issue.”

Validated live against one and two PRs, including complete pagination and stable JSON shape.

@ranadeepsingh Rana Singh (ranadeepsingh) changed the title docs: make the agent guide concise and actionable docs: add concise agent and PR readiness guidance Aug 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

## Summary
Refactor the Spark 4 branch references into a shared template plus concise spark4.0 and spark4.1 overlays derived from PRs 2646 and 2645. Preserve their core toolchain, codegen, R, Databricks, Fabric, CI, failure-triage, and porting differences.

## Prompting Intent
The engineer asked to validate that the new branch-specific skills are templatized versions of PRs 2645 and 2646, while retaining the important differences so those sync PRs can be updated later.

## Linked Sources
- Pull request: #2649
- Spark 4.1 sync/context: #2645
- Spark 4.0 sync/context: #2646

## Rationale
Extract shared Spark 4 responsibilities once, keep branch-only facts in small overlays, and add a reusable branch-reference template. This retains the operational knowledge from the long branch manuals without duplicating hundreds of lines or treating snapshot values as permanent truth.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 03:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/skills/synapseml-pr-loop/scripts/Get-PrReadiness.ps1:31

  • $Repo isn’t trimmed before splitting/validation, so inputs like "microsoft/SynapseML " (trailing space) will pass the current owner/name shape check but then fail later with a confusing API error. Trimming early makes the validation actually enforce the intended contract and improves diagnostics.
$repoParts = $Repo.Split("/")
if ($repoParts.Count -ne 2 -or -not $repoParts[0] -or -not $repoParts[1]) {
    throw "Repo must use owner/name format; got '$Repo'."
}
  • Files reviewed: 15/15 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

## Summary
Rename the skill to synapseml-branches, normalize references to branch-spark3p5/branch-spark4p0/branch-spark4p1, and explicitly map both master and spark3.5 to the Spark 3.5 context while preserving their different sync policies.

## Prompting Intent
The engineer requested branch-oriented filenames without dots, compliant Agent Skills naming/frontmatter, and a clear mapping in the skill showing that master currently uses the Spark 3.5 baseline.

## Linked Sources
- Pull request: #2649
- Spark 4.1 context source: #2645
- Spark 4.0 context source: #2646

## Rationale
Use predictable branch-<runtime> filenames, pluralize the routing skill because it covers multiple targets, and share one Spark 3.5 reference while distinguishing canonical master development from the shared spark3.5 release branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 04:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 14/14 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread AGENTS.md
- Record *why* a divergence exists at the point it is introduced — in a comment
next to the change and, if it is durable, in `AGENTS_<branch>.md`. A pin with
no rationale gets "helpfully" reverted by the next sync.
Black is pinned in [pyproject.toml](pyproject.toml); use that version.
I documented the opposite earlier today, and the pipeline trigger has since
been fixed, so the guidance is now wrong in the direction that costs the most:
an agent reading it would skip /azp run entirely and hand-queue every build.

Measured before changing the text. Commenting /azp run on the two port-branch
PRs produced builds 231455958 and 231455959, both recording reason=pullRequest
and requestedFor=GitHub, where every build queued by hand beforehand recorded
reason=manual under a personal account. The definition's pullRequest trigger
filter now reads +master | +spark3.5 | +spark4.0 | +spark4.1.

That reason field is the part worth writing down. A trigger-driven build and a
hand-queued one are otherwise indistinguishable in the UI, so it is the cheapest
way to answer "did my comment actually do anything" -- and the skill already
insists a comment is not evidence a build ran.

The UI-overrides-YAML explanation stays, demoted from cause to diagnostic: it
remains the first thing to check when a comment produces no build, because that
failure mode is completely silent -- no error, no build, no feedback anywhere.
The merge-ref fallback stays for that case, along with the warning against
queueing refs/heads/<branch>, which validates the branch rather than the merge
result.

The branch reference previously hedged with "historically did not queue ...
verify live behavior". Now that it has been verified, it states the current
filter and the date, so the next reader does not have to re-derive it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 17, 2026 04:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/skills/synapseml-pr-loop/SKILL.md:39

  • The instruction/link text suggests running scripts/Get-PrReadiness.ps1, but there is no repository-root scripts/Get-PrReadiness.ps1 (the script lives under .github/skills/synapseml-pr-loop/scripts/). As written, readers following the doc from the repo root will get a "file not found" and may assume the readiness tooling is missing.
- Run
  [scripts/Get-PrReadiness.ps1](scripts/Get-PrReadiness.ps1)
  with `-PullRequest <numbers>` and retain its JSON locally as the initial
  snapshot. It can contain review text; redact it before public sharing.
  • Files reviewed: 14/14 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@ranadeepsingh
Rana Singh (ranadeepsingh) merged commit c2f4ce7 into master Aug 17, 2026
14 checks passed
@ranadeepsingh
Rana Singh (ranadeepsingh) deleted the docs/agents-branch-model branch August 17, 2026 05:27
Rana Singh (ranadeepsingh) pushed a commit that referenced this pull request Aug 17, 2026
…ills

Brings in #2649, which rewrote AGENTS.md and CONTRIBUTING.md and added the
synapseml-branches and synapseml-pr-loop skills.

AGENTS.md conflicted because both sides rewrote it. Resolved by taking master
wholesale rather than merging line by line: the file is required to stay
byte-identical on every branch, so any blended result would be wrong by
construction. Verified by blob hash after resolving, not by reading the diff.

Nothing was lost in that resolution. The branch material this file used to
carry -- the branch table, the sync-and-conflict rules -- now lives in the
synapseml-branches skill, which arrives in this same merge.

CONTRIBUTING.md merged cleanly and is likewise identical to master. All twelve
new skill files arrived unchanged.

synapseml-local-setup still differs from master, and should. #2649 never
touched it; the divergence predates this merge and is version-driven -- JDK 17
and Scala 2.13 against master's 11 and 2.12. Taking master's side would have
handed this branch a toolchain it does not use. Confirmed by diffing the merge
base against master for that path and finding it empty.

AGENTS_spark4.0.md is untouched here and its future is a separate question:
master's new model routes branch facts to the branch skill, and the condensed
branch-spark4p0.md reference was derived from this very file, so the two now
overlap. Consolidating is a content decision, not a merge decision, and is
deliberately not bundled into a sync commit.

Docs only. No Scala, Python, build, or pipeline changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh) pushed a commit that referenced this pull request Aug 17, 2026
…ills

Brings in #2649, which rewrote AGENTS.md and CONTRIBUTING.md and added the
synapseml-branches and synapseml-pr-loop skills.

AGENTS.md conflicted because both sides rewrote it. Resolved by taking master
wholesale rather than merging line by line: the file is required to stay
byte-identical on every branch, so any blended result would be wrong by
construction. Verified by blob hash after resolving, not by reading the diff.
The spark4.0 sync resolved the same conflict the same way, and both branches
now carry the identical blob.

Nothing was lost in that resolution. The branch material this file used to
carry now lives in the synapseml-branches skill, which arrives in this merge.

synapseml-local-setup still differs from master, and should. #2649 never
touched it; the divergence predates this merge and is version-driven, so
taking master's side would have handed this branch a toolchain it does not use.

AGENTS_spark4.1.md is untouched. Master's new model routes branch facts to the
branch skill, and branch-spark4p1.md overlaps this file, so consolidation is a
content decision and is deliberately kept out of a sync commit.

Docs only. No Scala, Python, build, or pipeline changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh) added a commit that referenced this pull request Aug 17, 2026
* docs: add the Spark 4 branch details the condensed skill dropped

The branch references in #2649 were condensed from the branch guides in #2645
and #2646, and the condensation kept the conclusions but dropped a few facts
that are only useful in their specific form. Adding those back before the
source guides are removed from the port branches, so nothing is lost when they
go.

pyarrow and mlflow move together. The pinned MLflow requires pyarrow<20, so
raising pyarrow alone breaks the environment solve -- a failure that surfaces
during dependency resolution, far from the pin that caused it. Both comments
already say so in environment.yml; the skill said only "preserve dependency
comments", which does not tell you the two are coupled.

RCodegenSuite exists on both Spark 4 branches, not just 4.1 where the reference
happened to mention it. That asymmetry mattered: I had it recorded as missing
from 4.0 and worth back-porting, and checking the tree rather than trusting the
note showed it was already there. It turns an R failure into a unit test instead
of a full pipeline run, so it belongs in the shared file where both branches see
it.

DatabricksCPUStreamingTests is unscheduled on both branches -- confirmed absent
from both pipeline.yaml files, not merely undocumented. Only the 4.1 reference
mentioned it, which reads as a 4.1 quirk rather than a coverage gap both
branches carry.

The 4.0 reference also claimed the branch "historically had no PR checks even
when master contained corrected filters". That was true when written and is now
misleading: the ADO trigger filter was widened on 2026-08-17 and /azp run queues
this target, verified by build 231455959 recording reason=pullRequest. The
mechanics live in the common file, so this now points there and keeps only the
instruction that still matters -- confirm a build actually queued.

Finally, "two of four GPU notebooks failing" is recorded as the gap's expected
shape, because checking the count distinguishes the known gap from a new
regression where red/green cannot. The petastorm shim is marked explicitly
unproven so it stops being repeated as the cause.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct the Petastorm shim as a pyarrow gap, not a Python 3.13 one

Copilot flagged that the spark4.0 GPU bullet read as if spark4.1's shims caused
spark4.0's failure. Checking the trees to reword it turned up something larger
than the wording.

_petastorm_compat.py has no Python version gating anywhere in its 510 lines.
What it does is reimplement pyarrow APIs that Petastorm still calls and pyarrow
no longer ships -- ParquetDataset, pyarrow.filesystem, pyarrow.hdfs, dataset
pieces and partitions. Both Spark 4 branches pin pyarrow==18.0.0. So this is a
library-version problem, and spark4.0 is not exempt from it by being on an
older Python.

Both references said otherwise. spark4.0's told readers not to copy "Python 3.13
petastorm/cloudpickle shims" and listed them as do-not-port; spark4.1's grouped
them with genuine 3.13 concerns. That framing makes the layer look inapplicable
to 4.0 for a reason that is not true of it, which is worse than saying nothing:
it forecloses the question.

The concrete divergence is now stated plainly. spark4.0 has _horovod.py but not
_petastorm_compat.py, so it uses the plain Horovod SparkBackend where 4.1
substitutes a Petastorm-compatible subclass.

Its unit tests do not cover this. Without a usable Horovod the estimators fall
back to stubs, so the Petastorm path never executes -- which is why the branch
can look healthy while missing the layer.

That also sharpens the GPU note Copilot was reading. The wheel is the first
blocker and it masks this one, so fixing the wheel alone should not be expected
to turn those notebooks green. The shims are removed from spark4.0's do-not-port
list and marked a back-port candidate instead, explicitly requiring validation
on real 4.0 rather than adoption on suspicion.

Docs only. Whether to port the layer is a separate change with its own
validation, deliberately not bundled here.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: link the cross-references instead of naming them in prose

Both new pointers to branch-spark4-common.md were plain text while the file's
own "Read ... first" line at the top is a link, so the same target was styled
two ways in one document.

Made them links. Verified all four relative targets in the references directory
resolve on disk rather than assuming a same-directory filename is safe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: say pins the same pyarrow version, not runs the same pyarrow

"Runs the same pyarrow" left it ambiguous whether the comparison was the
version, the runtime, or the branch. The claim only holds because both branches
declare pyarrow==18.0.0, so the sentence now says pins, and names the version as
the thing being compared.

Raised as a suppressed review comment rather than a posted one, so it had no
thread to reply to.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: make suppressed comments findable and gate review on the current head

The PR loop already said to read suppressed comments, three separate times, and
I still missed one. Saying it again would not have helped, because the skill
never said where they are.

Suppressed comments are not review threads. They live inside a collapsed section
of the review body, so querying reviewThreads returns zero while they exist, and
an agent doing exactly what the skill asked gets a clean result and moves on.
They also have no thread to reply to or resolve, which the skill never mentioned
either -- so the natural next step after finding one silently fails. The skill
now says all of that and directs the fix into the follow-up commit message or a
PR comment.

The second failure is timing. Automated review re-runs per commit and is
asynchronous, so auditing immediately after a push reads the review of the
previous head. It reports zero findings for code nobody has looked at yet, and
that is indistinguishable from being genuinely clean.

Get-PrReadiness.ps1 now makes that distinction visible rather than leaving it to
judgment: automatedReviewCoversHead compares the newest automated review's
commit against the current head and gates `complete`, and
suppressedReviewBodiesForHead narrows suppressed feedback to that head so stale
entries from earlier commits stop reading as outstanding.

Running it against the live PRs immediately justified the change. #2650 reported
one suppressed body but zero for the current head -- correctly stale, already
fixed. #2646 reported automatedReviewCoversHead false with no automated review
commit at all, which is the real state: that PR has zero reviews. Its clean
thread count was never evidence of anything. Under the old output both PRs
looked equally clean.

The readiness gates now say suppressed feedback must be read from the review
body for the current head, and that coverage is compared by commit rather than
by recency, since a review produced before the last push never saw that code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: match the review bot by login instead of by substring

Review caught a real defect in the coverage gate I had just added. Automated
reviews were identified with `-imatch 'copilot'`, a substring test against the
author login, so any account whose name merely contains the word counted as
automated coverage.

That is the exact failure the gate exists to prevent. A review from `copilotfan`
would have set automatedReviewCoversHead true and marked the PR complete while
the actual reviewer had never seen the head commit -- a false all-clear produced
by the check meant to catch false all-clears.

Matching is now an exact comparison against a configurable login list, with the
`[bot]` suffix normalised so both `copilot-pull-request-reviewer` and
`copilot-pull-request-reviewer[bot]` match the same entry. The list is exposed
as -AutomatedReviewer so a repo using a different reviewer does not have to edit
the script.

The matched login is also emitted as latestAutomatedReviewAuthor, so a wrong
match is visible in the output rather than hidden behind a boolean.

Verified against the three real logins and four adversarial ones: `copilotfan`,
`my-copilot-bot` and `not-copilot` all matched under the old test and none match
now, while the genuine bot still does with and without the suffix. Live output
for #2650, #2646 and #2645 is unchanged, each attributed to
copilot-pull-request-reviewer on its current head.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: emit a readiness snapshot instead of throwing under StrictMode

Suppressed review comment on ebe6377 flagged that $latestAutomated can be
null when a PR has no automated reviews, and that reading .commit.oid off it
would throw and stop the script emitting a snapshot at all.

The stated failure does not reproduce. The script sets no StrictMode, so null
property access coalesces rather than throws; run against a PR with no matching
automated review it emits covered=False, complete=False and exits cleanly. That
path was already exercised for real, when #2646 had zero reviews.

The underlying concern is real though, just not where it was reported. With
StrictMode enabled by the caller -- which propagates into this script's scope --
it does throw, and the first failure is not this line. It is $response.errors in
the GraphQL error check, then .state, with the null review fields never reached.
It also throws on the completely normal path with a real review present, so this
was never about null reviews: the script has never been StrictMode-safe anywhere.

Fixing that one reported line would have produced a partial fix wearing the
appearance of a complete one, so the contract is now explicit instead. The
script pins Set-StrictMode -Off for its own scope, so its behaviour no longer
depends on the caller's session state, and it emits a snapshot describing what
is missing rather than failing to report at all -- which is the outcome the
comment asked for, and now holds on the normal path too.

The three review fields are still made explicitly null-safe. They no longer rely
on null-coalescing to do the right thing, so the intent is readable rather than
incidental.

Verified in both modes: StrictMode 3.0 with a forced-null reviewer gives
covered=False complete=False, StrictMode 3.0 on the normal path gives
covered=True complete=True attributed to copilot-pull-request-reviewer, and both
threw before this change. Default-mode output for #2650, #2646 and #2645 is
unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: ground the pyarrow bound and the streaming gap in checkable facts

Reviewing this PR's own claims against the branch trees turned up two that were
weaker than they read.

The pyarrow/mlflow bullet said "the pinned MLflow requires pyarrow<20" and told
the reader to check the inline comments in environment.yml. Those comments
disagree: spark4.0 says pyarrow<20, spark4.1 says pyarrow<19. mlflow 2.21.3's
published metadata declares pyarrow<20,>=4.0.0, so spark4.0 is right and 4.1's
comment is wrong -- and the guidance as written would have sent someone on 4.1
to a comment that misinforms them. It now states the bound from the package
metadata and flags the stale comment instead of pointing at it. Fixing 4.1's
comment belongs on that branch, not here.

The streaming bullet recorded that DatabricksCPUStreamingTests is unscheduled
and that scheduling it needs pool capacity and a notebook fix, which is faithful
to the note it came from but stops at status. Verified the mechanism: the class
is real in DatabricksCPUTests.scala, pipeline.yaml names its CPU legs explicitly
as DatabricksCPUTests1..5 plus DatabricksGPUTests and never lists it, and it is
a separate class because the streaming notebook's server.stop() cancels
concurrent SparkContext jobs, so it needs its own cluster rather than a slot on
an existing leg. That is why capacity is the blocker, which the bullet asserted
without explaining.

The first draft of that sentence generalised the in-repo comment from "Spark
4.0" to "Spark 4". Both branches carry the same comment naming 4.0, and nothing
re-confirms it on 4.1 -- which is the same overgeneralisation that produced the
Petastorm error this PR exists to correct. It now reports what the comment says
and that 4.1 is unverified.

Also verified and left unchanged: RCodegenSuite.scala is present on both
branches, and all relative links still resolve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: wait for the automated review of the current head instead of sampling once

The loop kept reporting a clean PR seconds after a push, before review of that
push existed. Nothing was wrong with the query; it answered honestly about a
head nobody had reviewed yet. The gap was that the result was read as clean, so
findings surfaced only when a human pointed out that comments had appeared.

Review is triggered automatically on push, so the fix is not to ask for one. It
is to wait for it. -WaitForReview polls until the newest automated review's
commit matches the current head, with -TimeoutMinutes and -PollSeconds to bound
it, and on timeout it warns that findings may still be pending rather than
returning a quiet all-clear. The per-PR body moved into Get-PrSnapshot so the
wait re-queries real state each time instead of re-reading a stale snapshot.

This puts the waiting inside the tool. It was previously prose in the skill, and
prose asking an agent to hand-roll a polling loop is exactly what got skipped --
the same loop has now been written by hand three times, which is evidence the
instruction does not survive contact.

-RequestReview stays but is demoted to a fallback and documented as one. It
exists because #2646 was found with zero reviews, where waiting alone would
block until timeout, and the REST handle is the only path that works: the
GraphQL requestReviews mutation rejects the reviewer's Bot node id and
`gh pr edit --add-reviewer` cannot resolve the bot login. The reviewer never
appears in requested_reviewers afterwards either, so a request cannot be
confirmed by reading that list back -- only by waiting for coverage, which is
now what the script does.

Verified: without -RequestReview no request is sent; with it, and coverage
false, one is. Both StrictMode and default-mode paths still emit a snapshot, and
output for #2650/#2646/#2645 is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: recover the last branch facts and bound the pyarrow claim correctly

A gap audit of both AGENTS_<branch>.md files against the skill references, run
specifically to find what deleting them would destroy, came back with four
durable facts that exist in no reference file. Each is verified against the
branch trees rather than copied on the strength of the note it came from.

cyber/utils/spark_utils.py differs between the branches -- 4.0 builds its
indexed frame with rdd.toDF(schema), 4.1 with spark.createDataFrame(rdd,
schema) -- and toDF was measured working on both 4.0.1 and 4.1.1. Confirmed both
forms in the tree, and confirmed that df.rdd.zipWithIndex() still precedes both,
which is what makes the back-port low-value: it removes one monkey-patched RDD
call and leaves another. Without this, the difference reads as a version
requirement and gets preserved forever or "fixed" for no gain.

The r-base=4.4 pin was recorded on neither reference, although the sparklyr
1.9.5 pin next to it was. Both branches pin r-base=4.4, and the 69/69 RTests
result was measured for the pair, so carrying one half invites someone to move
the other.

Fabric re-enablement kept the intent but lost the specifics: the payload lives
in the Fabric test package's FabricOperations.scala, hardcodes
'SparkVersion': '3.5', and must request '4.1'. Verified that master, spark4.0
and spark4.1 all carry that same hardcoded value, which is the point worth
recording -- it means editing it on the port branch cannot affect master, so an
agent has no reason to refuse the change as cross-branch. The
sempy-integration-region capacity requirement is the one prerequisite that
cannot be discovered from code at all.

The sparklyr 1.9.3 failure signature is now written down. The reference already
said to read the backtrace and not to blame a dead session; it did not say what
to look for. The frame chain through dbplyr:::select.tbl_lazy,
sparklyr:::tidyselect_data_proxy.tbl_spark and simulate_vars_spark surfaces as
invoke_static/hive_context on NULL, which is exactly why it reads like a dead
session.

Separately, a suppressed comment on b61db32 was right about the lockstep
bullet. "Raising pyarrow alone breaks the environment solve" overstates it: the
declared bound is pyarrow<20, so 18 to 19 is fine and only crossing the bound
without moving mlflow breaks. Now bounded as stated rather than as remembered.

Deliberately not carried over, per the audit: the claim that RCodegenSuite does
not exist on spark4.0, which is false and contradicted by the tree, and the
framing of the Petastorm shims as Python 3.13 workarounds, which is the error
this PR exists to correct.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: state the pyarrow/mlflow rule instead of a branch state that is not live yet

Review caught the lockstep bullet asserting `mlflow==2.21.3` with `pyarrow<20`
as though it described both Spark 4 branches. It does not describe either one as
they exist today.

I read those pins out of the sync worktree, which is the result of #2646 rather
than the live branch. The live pins are not close to what the bullet implied:
master has pyarrow 10.0.1 with mlflow 2.21.3, spark4.1 has 18.0.0 with 2.21.3,
and spark4.0 has pyarrow 22.0.0 with mlflow 1.26.1 -- a different MLflow major
with a different bound, and a pyarrow above the number the bullet said could not
be crossed. On master, where this file lands, the claim would have been wrong for
every branch a reader might check.

That is the third time in this work that a derived tree has been written down as
current fact, so the bullet no longer states pins at all. It states the rule --
the bound comes from the pinned MLflow, check that version's own metadata, read
both live values on the branch being edited -- and keeps 2.21.3's pyarrow<20 as
a worked example rather than a claim about the world. That stays true after
#2646 merges and after the next pin bump, which the previous wording would not
have.

Also fixes a suppressed comment on the same head: $automatedLogins rejected
empty strings but accepted whitespace-only ones, since ' ' is truthy in
PowerShell. That produced a non-empty login list guaranteed never to match,
leaving automatedReviewCoversHead false forever with no error to explain why --
a silent hang in the gate that exists to stop silent passes. Entries are now
trimmed and blank ones dropped before the emptiness check.

Verified: '  ' and @('  ','') both throw with a clear message, @('  ','Copilot')
is accepted, and default behaviour is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct branch claims verified against live refs

Addresses the suppressed Copilot comment on branch-spark4-common.md:69 and a
wider audit it prompted. Every branch claim below was re-verified with
`git show ms/<branch>:<path>` / `git grep <pat> ms/<branch>` against the live
refs rather than against local sync worktrees, which are PR *results* and do
not reflect current branch state. That contamination is what produced the
wrong claims being fixed here.

- DatabricksCPUStreamingTests: was "unscheduled on both Spark 4 branches".
  Live spark4.0 schedules it (job databricks-cpu-streaming); the class does
  not exist on master at all. Rewritten to send readers to pipeline.yaml on
  their own branch, and to note that dropping the leg while leaving the class
  defined is silent coverage loss.
- RCodegenSuite: was "on both Spark 4 branches". Live spark4.0 does not have
  it; spark4.1 does.
- sparklyr: was "both branches pair that pin with r-base=4.4". Live spark4.0
  is on 1.9.3, spark4.1 on 1.9.5, so the 1.9.3 backtrace signature documented
  below it is a live concern on 4.0, not history. r-base=4.4 on both is
  correct and is retained.
- pyarrow: branch-spark4p0.md still claimed "both branches pin the same
  pyarrow" - the same error already corrected in the common doc. Live
  spark4.0 is 22.0.0, spark4.1 is 18.0.0. The Petastorm conclusion is
  unchanged and in fact strengthened: 4.0 resolves to the newer pyarrow, so
  the APIs the shim restores are at least as absent there.
- NumPy: was "Keep NumPy 1.26.4 pinned", and separately listed "unpinned
  NumPy" as 4.1-only. NumPy is unpinned on both live branches; the pin
  described post-merge state. Restated as a conditional rule.

Also retracts an earlier unverified suspicion of mine: LongOffset is present
on live spark4.0 (DistributedHTTPSource.scala, HTTPSource.scala). The empty
grep that prompted it was a case-sensitivity error, not a defect.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: preserve branch operating detail; fix review-coverage by commit

Addresses the suppressed Copilot comment on Get-PrReadiness.ps1:251 and closes
the content-loss gaps found by auditing the skill against the files it is
meant to replace.

Review coverage:
  automatedReviewCoversHead was computed from the newest automated review
  overall, so a force-push back to an already-reviewed commit reported a
  reviewed head as uncovered and would send -WaitForReview into a pointless
  wait. Coverage is now true when *any* automated review targets the head, and
  the reported review prefers one that covers the head, falling back to the
  newest for diagnostics.

Knowledge preservation:
  AGENTS_spark4.0.md and AGENTS_spark4.1.md exist only on the two sync
  branches; neither live spark4.0 nor live spark4.1 carries any branch
  documentation. Deleting them therefore destroys that knowledge unless this
  skill carries it, so the condensation was re-audited line by line against
  both sources. Recovered here:

  - generated Python lives under target/scala-2.13, not master's scala-2.12
  - the Java 17 surface: Dockerfile JAVA_HOME, pr-validation.yml JDK, and the
    exact CMS flags whose restoration stops the JVM booting
  - why environment.yml pins move (pip too old, first torch/torchvision
    releases, interpreter-specific wheel URLs) and why their comments matter
  - the Spark 4 adaptations by name: UnboundRowEncoder and the SAR case class,
    DetectAmbiguousSelfJoin and the qualified join column, safeGetDefault, and
    why the classifier fixture dropped NaN rather than weakening the assertion
  - OpenAIPromptPythonOverrides.scala and the NameError that zero-arg super()
    avoids
  - the __init__.py rationale and its guard tests (test_http_package.py,
    test_package_exports.py)
  - rLoadLine/new_ml_pipeline_stage: signature stable v1.8.0-v1.9.5, four
    param types share one implementation, keep the three assertions in step,
    and it is alignment rather than a proven fix
  - the sibling-diff habit with its command and its track record
  - GPU pool name, its 1 x 3 worker shape, and the sibling branch as a free
    control; CPU pool name on 4.1
  - areLibrariesInstalled == false is a ~10 minute timeout (60 * 10 attempts)
    and inverts expectations: a genuine FAILED throws instead
  - the torchvision 0.17.0 / torch 2.2.0 downgrade instance
  - the fine-tune measurements showing the wheel swap was alignment, not a fix
  - Fabric on 4.0 may never be possible; supersession by 4.1 is likelier

  Also corrects branch-spark4p1.md, which still claimed spark4.0 "pins the
  same pyarrow version". It does not - it pins a newer one, which strengthens
  rather than weakens the point that spark4.0 is not exempt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: scope Spark 4 claims to sync-PR state, correct GPU pool

A third pass of verifying claims against live refs found that most of
AGENTS_spark4.0.md describes the branch *after* #2646, not as it stands. The
guides were written on the sync branches, so they describe the merged result;
importing them as present-tense fact overstates what live spark4.0 has.

Measured against ms/spark4.0 (git ls-tree / git grep, not a worktree):

  - GPU pool is synapseml-build-17.3-gpu, not the shared synapseml-build-14.3-gpu.
    #2646 switches it to the shared pool, so "the pool is shared, use the sibling
    branch as a control" is a post-merge property, not a current one.
  - OpenAIPromptPythonOverrides.scala: absent
  - test_http_package.py / test_package_exports.py: absent, so the __init__.py
    policy is currently unenforced there
  - new_ml_pipeline_stage in generated R: absent

Changes:
  - the header now states the guides describe the branches as of #2645/#2646,
    lists what live spark4.0 lacks, and names the worktree-vs-live mistake
    explicitly so the next reader does not repeat it
  - the GPU bullet no longer hardcodes a pool name; pool names differ by branch
    and the sharing-derived advice is scoped to where sharing actually holds
  - the guard-test, OpenAIPrompt and rLoadLine bullets say where they apply
  - branch-spark4p0.md records the live-vs-#2646 delta directly

Verified correct and left as-is: CMS flags absent on both Spark 4 branches and
present on master; spark4.1's CPU pool synapseml-build-18.0 and shared
14.3-gpu; GpuConcurrentRuns; areLibrariesInstalled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: scope StrictMode to functions; narrow automated-reviewer default

Addresses both suppressed Copilot comments on 7577faa.

Get-PrReadiness.ps1:40 - the default reviewer list contained "copilot" and
"github-copilot". Matching is exact, but those shorter logins are registerable
by humans, so an exact match could count a human review as automated coverage
and set complete=true on an unreviewed head. The default is now this repo's
reviewer bot alone (copilot-pull-request-reviewer); -AutomatedReviewer still
extends it for other repos.

Get-PrReadiness.ps1:69 - Set-StrictMode -Off sat at script scope, so
dot-sourcing disabled StrictMode for the caller's whole session. Rather than
document that hazard, it is now scoped to the two functions that actually read
sparse GraphQL fields positionally (Invoke-PagedQuery, Get-PrSnapshot). Script
scope only validates parameters and defines here-strings, so it does not need
the relaxation.

That move initially introduced a silent defect worth recording: Set-StrictMode
was placed *before* each function's param() block. PowerShell requires param()
to be the first statement in a function body, and when it is not, param(...)
parses cleanly as a command invocation and fails only at runtime. The file
parsed clean while being broken, so parse-checking was not sufficient
verification here.

Verified by running, not by inspection:
  - normal invocation reports covers=True, attributed to
    copilot-pull-request-reviewer, so narrowing the default did not break
    detection
  - a caller with Set-StrictMode -Version 3.0 still gets a snapshot, which is
    the original defect this relaxation exists for
  - after dot-sourcing, a missing-property access in the caller still throws,
    confirming StrictMode is no longer disabled session-wide

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: count only submitted reviews as head coverage

Addresses the review comment on Get-PrReadiness.ps1:259.

$reviewsForHead matched on commit.oid alone. A pending review already carries
the head's commit oid but has a null submittedAt, so an in-progress review
would have satisfied automatedReviewCoversHead - reporting complete=true and
releasing -WaitForReview while the review was still being written. That is the
same premature all-clear this gate exists to prevent, so the filter now
requires submittedAt as well as a matching oid.

This is the third defect of this class found in this gate: substring login
matching, then coverage decided by recency instead of commit, now coverage
satisfied by an unsubmitted review. Each one made the check answer "yes" when
the honest answer was "not yet".

Verified: a live run still reports covers=True on the real head, and a
simulated pending review carrying the head oid with a null submittedAt yields
coversHead=False.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: make complete mean complete; request a review at most once

Addresses the review comment on Get-PrReadiness.ps1:293, plus a worse defect
that comment exposed.

The reported defect: under -WaitForReview, Get-PrSnapshot runs once per poll,
so -RequestReview re-POSTed the reviewer request every cycle until coverage
arrived - notification spam and a rate-limit risk. The request is now attempted
at most once per PR per invocation, with the outcome recorded and reused.

The defect that surfaced alongside it: completeness.complete tested
$truncatedThreadComments, which counts comment-pagination truncation, not
unresolved review threads. The two were conflated, so the field reported
complete=true on this very PR while a review thread was open - the headline
readiness signal giving an all-clear over outstanding review feedback. It now
requires all of: no pagination truncation, an automated review covering the
head, and zero unresolved threads, suppressed-for-head items, failed checks and
pending checks. Pending is treated as unknown rather than passing.

Verified against live #2650 with one thread open: the same input that
previously produced complete=true now produces complete=false.

readiness-gates.md documents what the flag now means, and says to trust the
individual fields over the summary when they disagree - that flag has now been
wrong in both directions.

This is the fourth false all-clear found in this gate (substring login
matching, coverage by recency, unsubmitted reviews counting as coverage, and
now unresolved threads not counting at all). Every one resolved in the
direction of declaring readiness that had not been established, which is worth
recording as a property of the tool rather than four separate slips.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: restore the remaining searchable specifics from the branch guides

Knowledge transfer was measured rather than assumed: 40 distinctive terms from
AGENTS_spark4.0.md and AGENTS_spark4.1.md were tested against the skill, and
six were absent. Five are restored here because each is the string an engineer
would actually search for when hitting the failure:

  - spark.sql.ansi.enabled / spark.sql.ansi.doubleQuotedIdentifiers, and
    PARSE_SYNTAX_ERROR as the symptom when the second flag is missing
  - AdbGpuRuntime as the field that must not be bumped to DBR 18, with the
    reason (17.3 LTS ML ships Spark 4.0, 18.0 ML ships 4.1, so bumping makes
    the suite green by no longer testing this branch)
  - refs/pull/<N>/merge as the literal ADO fallback target, and why
    refs/heads/<branch> fails (service-connection authorization)
  - ImageFeaturizerSuite as the ONNX OOM site, and the conda HTTP 403 in
    RTests vw

The sixth, cloudpickle, is deliberately not restored. Both guides describe the
Petastorm shim as working around "cloudpickle/petastorm breakage under Python
3.13". Reading the file disproved that: it contains no Python-version gating at
all and reimplements pyarrow APIs Petastorm still calls. Restoring the term
would restore the error, and the corrected framing is already recorded.

Coverage after this change: 39/40, with the one omission reasoned above.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: gate readiness on required checks being present, not just green

A head whose Azure Pipelines build never queued reported failed=0 and
pending=0 and scored complete, because an absent check is neither. On
this pull request the GitHub Actions checks started 12s after the push
and all passed, while the ADO build only appeared 3m17s later and only
because a human commented `/azp run`. Between those points every
mechanical gate read clean on a head that had no CI on it.

Add `missingRequiredChecks` (prefix-matched, default `microsoft.SynapseML`)
to the snapshot and to `complete`, extend `-WaitForReview` to wait for
required checks as well as the review, and add `-RunPipeline` to post
`/azp run` when the build is missing -- guarded to at most one comment per
PR per invocation, mirroring `-RequestReview`, since the wait loop calls
the snapshot every poll.

Document that the build does not re-queue itself on a push, so every push
needs its own comment, and that green Actions checks are not CI passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: make the spark4.0 GPU pool claim self-verifying

Addresses both suppressed review comments on 0f2f9bd, which reported
the `synapseml-build-17.3-gpu` claim as stale. Measured instead of taken
on trust, and the claim holds -- `git show` of
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala:

  ms/master                      14.3-gpu  / 14.3.x-gpu-ml-scala2.12
  ms/spark4.0 (live)             17.3-gpu  / 17.3.x-gpu-ml-scala2.13
  ms/spark4.1                    14.3-gpu  / 18.0.x-gpu-ml-scala2.13
  ms/sync/spark4.0-with-master   14.3-gpu  / 17.3.x-gpu-ml-scala2.13

The reviewer read the sync branch as live `spark4.0` -- the same
worktree-for-branch confusion this paragraph exists to warn about, made
against the paragraph itself. So the wording was not wrong, it was not
checkable: cite the file and the value, and the next reader can repeat
the check in one command rather than deriving it from whichever tree is
open.

Record what #2646 actually changes here, since it is operating knowledge:
the GPU pool moves to master's shared `14.3-gpu` resolved by
`getPoolIdByNameAndNodeType`, while the CPU pool stays 17.3 and the GPU
runtime stays 17.3.x. `spark4.1` already pairs `14.3-gpu` with an 18.0.x
runtime, so the pool name identifies a warm node pool, not a DBR version.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants