fix(core): include reference-only distribution categories - #2630
Open
Rana Singh (ranadeepsingh) wants to merge 1 commit into
Open
fix(core): include reference-only distribution categories#2630Rana Singh (ranadeepsingh) wants to merge 1 commit into
Rana Singh (ranadeepsingh) wants to merge 1 commit into
Conversation
Rana Singh (ranadeepsingh)
requested a review
from Brendan Walsh (BrendanWalsh)
as a code owner
August 14, 2026 20:55
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Collaborator
Author
|
/azp run |
|
Hey Rana Singh (@ranadeepsingh) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
August 14, 2026 20:56
View session
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes DistributionBalanceMeasure in the core exploratory module so that custom reference distributions correctly contribute categories that appear only in the reference map (not observed in the dataset), ensuring distance metrics (e.g., JSD) are computed over the full intended support. It also updates documentation to describe the new/clarified support semantics and adds regression tests covering reference-only and observed-only categories, key typing, validation, and persistence.
Changes:
- Materialize reference-only (positive-probability) categories into a small DataFrame, union with observed stats, and aggregate so metrics are computed on the union support.
- Add validation for reference probabilities (range + sum-to-1 tolerance) and key conversion/uniqueness for integral sensitive columns; improve numeric decoding robustness for persisted params.
- Expand Scala test coverage and clarify documentation for custom reference distribution behavior.
Show a summary per file
| File | Description |
|---|---|
| docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md | Clarifies custom reference distribution semantics, including union support behavior for observed-only and reference-only categories. |
| core/src/test/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasureSuite.scala | Adds regression tests for reference-only/observed-only categories, ordering/typing alignment, validation failures, and persistence. |
| core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasure.scala | Implements union-support alignment for custom references, validates distributions/keys, and ensures persisted numeric probabilities decode safely. |
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
## Summary Fix DistributionBalanceMeasure so custom comparisons cover the union of observed and positive-probability reference categories, with zero observed values for reference-only categories. Add validation, persistence coverage, focused regressions, and a semantics clarification. ## Prompting Intent Own GitHub issue microsoft#2010 end-to-end with a focused, scalable DataFrame fix. Preserve legacy outputs; cover reference-only and observed-only categories, ordering and supported types, zeros, invalid inputs, persistence, and generated APIs; validate only through the JDK 11 wrapper and the shared SBT mutex. ## Linked Sources - GitHub issue: microsoft#2010 - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale Materialize only the already-driver-resident reference parameter as a typed DataFrame, then union and aggregate it with distributed observed statistics. This aligns categories without collecting observed support or using RDD APIs. Positive-probability keys extend support, explicit zero entries behave like omitted keys, and typed-key validation prevents ambiguous numeric categories while preserving the public API and output schema. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
force-pushed
the
copilot/issue-2010-reference-categories
branch
from
August 14, 2026 21:42
273536a to
1642707
Compare
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
August 14, 2026 21:42
View session
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 16, 2026
## Summary Strengthen Jensen-Shannon distance verification with exact-value, symmetry, normalization, cancellation, zero-support, bounds, schema, and end-to-end assertions. Clarify the mathematical domain and smoothing semantics in the docs, and remove a stale chart whose embedded values used the old natural-log scale. ## Prompting Intent Independently make GitHub PR microsoft#2631 merge-ready by auditing its full mathematical and runtime behavior, API compatibility, edge cases, related PR microsoft#2630 interactions, review feedback, checks, generated bindings, Spark compatibility, and downstream documentation. Fix every valid issue without bundling the separate reference-support implementation. ## Linked Sources - GitHub issue microsoft#2006: microsoft#2006 - Pull request microsoft#2631: microsoft#2631 - Related reference-support pull request microsoft#2630: microsoft#2630 - Copilot review feedback: microsoft#2631 (review) - Jensen-Shannon definition: https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence ## Rationale Keep the production correction narrowly scoped to dividing Jensen-Shannon divergence by ln(2), preserving natural-log KL divergence and all public APIs. Use independently calculated constants and tolerant assertions to validate semantics across Spark/JVM versions. Document that the unit bound assumes valid probability distributions and that midpoint mixing, not additive smoothing, handles zero support. Leave reference-only category materialization and custom-distribution validation to microsoft#2630 to avoid duplicating unrelated work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Coordination note for #2631 (
The shared |
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2010
Root cause
DistributionBalanceMeasurederived its category rows only from observed data, then looked up custom reference probabilities on those rows. A category that existed only in the reference map therefore never reached any metric aggregation, truncating the reference distribution.Change
Tests
All SBT commands ran through the JDK 11
synapseml-sbt.shwrapper while holding the named mutexSynapseMLSbtValidationfor the full command.core/testOnly com.microsoft.azure.synapse.ml.exploratory.DistributionBalanceMeasureSuite— 19/19 passed.core/testOnly com.microsoft.azure.synapse.ml.exploratory.AggregateBalanceMeasureSuite com.microsoft.azure.synapse.ml.exploratory.DistributionBalanceMeasureSuite com.microsoft.azure.synapse.ml.exploratory.FeatureBalanceMeasureSuite— 36/36 passed.core/compile— passed.scalastyleandtest:scalastyle— passed with zero findings.core/codegen— passed; Python and R wrappers generated.UDFTransformerSuitelocal smoke test — 1/1 passed.black --check --extend-exclude 'docs/' .— passed (189 files unchanged).Regression coverage includes reference-only and observed-only categories, mixed map ordering and integral/string types, explicit zero probabilities, invalid sums/probabilities/keys, persistence, and unchanged issue-example results.
Performance notes
The observed support remains fully distributed. The implementation does not collect observed categories or use RDD APIs; only the already driver-resident reference parameter is converted to a small DataFrame.
unionByNameplusgroupByperforms support alignment on Spark and retains the existing per-feature support-cardinality action.Review
The project
code-reviewchecklist was applied independently to numerical correctness, DataFrame scalability, API compatibility, serialization, and edge cases. No high- or medium-severity findings remained. A later automated review note about NaN test assertions was fixed, and the release-branch patch was adjusted to apply cleanly tospark4.1.