Skip to content

[Bug] Support schema-evolved multi-group Measure aggregation #14069

Description

@hanahmily

Summary

Make valid aggregate queries succeed when participating storage groups have different projected schemas. Follow-up to #14066 under #13998.

Size audit and dependencies

Classification: tracking parent, not an executable leaf or automated implementation task.
Boundary: multi-group DistributedPlan.Execute through executeAgg, ReduceRawFrames, and ReducePartialBatches.
Schema alignment and storage-group-aware replica handling are independently testable changes; split them into executable tasks before implementation.
Blocked by: #14066 via BanyanDB PR apache/skywalking-banyandb#1326, to finalize the remaining boundary after legacy removal.

Proposed initial slice

Normalize missing projected, non-grouping tags to null before partial reduction, rather than rejecting the query because partial layouts differ. Do not simply disable structural schema validation.

Concrete oracle

  • Old schema: svc,value; rows (old,2) and (old,3).
  • New schema: svc,zone,value; rows (new,west,7) and (new,west,11).
  • Query: GROUP BY svc, SUM(value), project svc,zone.

Expected unordered output:

old, NULL, 5
new, west, 18

Different service keys deliberately avoid ambiguity over representative values of non-grouping tags. Currently differently shaped partial batches can fail with partial ... schema mismatch.

Completion criteria

  • The schema-evolution oracle succeeds through the distributed production caller, not only a reducer helper; add a regression that fails on the current structural mismatch.
  • Normalization preserves aggregate-state columns, including count sidecars used by MEAN, and rejects genuinely incompatible types with a clear error.
  • Replica deduplication preserves storage-group provenance: contributions 5 and 18 for service api from two different storage groups must total 23, even when both groups use shard 0; duplicate replicas of either contribution must not increase that sum. This requires its own focused implementation/test slice.
  • Each executable child activates the real distributed caller and has independently specified expected results.

Relevant code: pkg/query/vectorized/measure/reduce.go, pkg/query/vectorized/measure/plan/distributed.go; extend distributed-plan and topology-matrix coverage.

go test ./pkg/query/vectorized/measure/...
go test ./test/integration/distributed/query/...

Out of scope of the first slice

Missing GroupBy keys: these fail earlier during planning and need a separately specified change, not an assumed reducer-only fix. Also exclude changing aggregate/operator semantics or replacing the ordinary multi-group row merger.

Parent: #13998.

Compatibility and review policy

  • Preserve existing on-disk and wire formats and mixed-version compatibility. Any necessary protocol change requires a separately reviewed compatibility contract.
  • Use explicit, hand-calculated expected results; do not depend on the removed row engine as the test oracle.
  • Human review enforces the no-new-row-path policy. A row-call-site lint guard is not required.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    databaseBanyanDB - SkyWalking native database

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions