Proposal: Generalize Collections into rule-defined subgraphs

[shangyian]

Summary

A collection today is a static list of nodes and is largely unused (it is not exposed appropriately in the UI). This proposes generalizing it into a rule-defined subgraph -- a small grammar that resolves a collection's membership (and, optionally, its join structure) from the graph, where a hand-curated list is simply the enumeration end of the same grammar. This unifies two recurring needs -- lineage-scoped node sets and curated join subgraphs -- under one primitive, and provides the scoping mechanism needed for subgraph-scoped search and exploration.

Motivation

Two distinct needs come up repeatedly and neither is served well today:

1. Lineage-scoped scoping/search: "Everything downstream of source/fact node X." Users want to search, filter, and browse within that subgraph without enumerating namespaces or downloading every node to filter client-side. Namespace-scoped search only helps when the region of interest happens to line up with a namespace.
2. Curated join subgraphs. A reviewed set of usable dimensions and join paths around some metrics -- this is a lens for discovery and exploration over the dimensional graph, where the value is the join edges/roles, not just the node set.

These map onto the two graphs overlaid on DJ nodes: the lineage DAG (noderelationship) and the dimensional join graph (dimensionlink / join-path resolution). A single rule-defined collection can express membership over either.

Current state

There are two tables related to collections:

collection        (id, name UNIQUE, description, created_by_id, created_at, deactivated_at)
collectionnodes   (collection_id, node_id)   -- many-to-many, explicit membership only

This setup supports only static, manually-populated node lists, has no rule/derivation concept, and no notion of edges/join paths.

Proposal

Add a definition to a collection describing how membership is resolved, using four composable knobs:

• seed: starting node(s)
• traverse(graph, direction, depth): expand over the lineage or join graph
• filter: restrict by node type / namespace / tag / role
• include / exclude: granular overrides at node and edge granularity

A fully hand-curated collection is one whose rule is pure include; an auto-updating one is seed + traverse. Hybrids (rule + manual tweaks) can also be supported.

Canonical model

class EdgeRef(BaseModel):
    from_: str = Field(alias="from")
    to: str
    via_role: str | None = None

class Filters(BaseModel):
    node_types: list[NodeType] = []
    namespaces:  list[str] = []
    tags:        list[str] = []
    roles:       list[str] = []          # join graph only

class MemberSet(BaseModel):
    nodes: list[str] = []
    edges: list[EdgeRef] = []            # edges only meaningful for graph: join

class Definition(BaseModel):
    graph:     Literal["lineage", "join"] | None = None   # None => static list
    seeds:     list[str] = []
    direction: Literal["downstream", "upstream", "both"] = "downstream"
    depth:     int | None = None                          # None => unbounded
    filters:   Filters = Filters()
    include:   MemberSet = MemberSet()
    exclude:   MemberSet = MemberSet()

Resolution semantics

1. nodes = filter(traverse(seeds, graph, direction, depth))
2. nodes = nodes ∪ include.nodes − exclude.nodes
3. If graph == "join": trace the usable join edges among nodes, then edges = edges ∪ include.edges − exclude.edges
4. graph absent ⇒ static: members are exactly include.nodes (today's behavior, unchanged)

Lineage traversal reuses get_downstream_nodes; join-edge validity reuses the existing join-path resolution. On refresh, every edge in include.edges is re-validated against the live graph. A curated edge whose join no longer resolves is dropped/flagged, so curated subgraphs can't silently retain broken joins.

Two views, and the relationship to cubes

• Lineage view (graph: lineage): node set only. Edges (dependency) are cheap and derived live. Membership is the artifact.
• Join view (graph: join): carries edges. Tracing the usable-join subgraph is expensive, so it's materialized (see below).
• A cube is the same definition grammar with curated join edges, plus query/materialization consumption semantics. Proposal is to unify the definition layer, not necessarily the artifact — a collection you search/navigate, a cube you materialize-and-query, both defined by this grammar.

Schema changes

ALTER TABLE collection      ADD COLUMN definition JSONB;                    -- NULL => static
ALTER TABLE collection      ADD COLUMN kind VARCHAR NOT NULL DEFAULT 'static';
ALTER TABLE collectionnodes ADD COLUMN source VARCHAR NOT NULL DEFAULT 'pinned';
  -- 'pinned'  = authored / manual include
  -- 'derived' = produced by resolving the rule

collectionnodes becomes the resolved-membership set. Refresh recomputes source='derived' rows only, so hybrids keep manual adds across refreshes. Reverse lookup ("which collections contain node X") stays a single indexed join. Existing static collections have definition = NULL and are unaffected.

Authoring format

• Storage & API: JSON (the definition column and request/response shape), validated by the model above.
• Authoring: JSON via UI/API for lightweight/ad-hoc collections; optional repo-backed YAML for durable/curated ones (review + audit, alongside node YAML). YAML deserializes into the same model. Not forced — no new mandatory authoring ceremony for the lightweight case.

Examples (YAML)

Dynamic lineage collection -- node set, no edges:

name: orders_downstream
description: All metrics/transforms downstream of the orders source.
definition:
  graph: lineage
  seeds: [default.orders]
  direction: downstream
  depth: null
  filters:
    node_types: [metric, transform]

Static curated list -- pure enumeration (today's behavior):

name: launch_dashboard
description: Hand-picked metrics for the launch dashboard.
definition:
  include:
    nodes:
      - default.total_revenue
      - default.total_quantity
      - default.avg_order_value

Curated join subgraph -- edges are the payload:

name: revenue_explore
description: Curated usable-join subgraph for revenue exploration.
definition:
  graph: join
  seeds: [default.total_revenue]
  direction: downstream
  depth: 2
  filters:
    roles: [order, customer]
  include:
    edges:
      - { from: default.order_details, to: default.date,     via_role: order }
      - { from: default.order_details, to: default.customer, via_role: customer }
      - { from: default.order_details, to: default.product }
  exclude:
    edges:
      # reachable via the customer hop but not meaningful for revenue
      - { from: default.customer, to: default.date, via_role: registration }

Hybrid -- rule + granular overrides:

name: orders_metrics_curated
description: Downstream order metrics, minus deprecated, plus one external metric.
definition:
  graph: lineage
  seeds: [default.orders]
  direction: downstream
  filters:
    node_types: [metric]
  include:
    nodes: [default.page_view_count]     # outside orders lineage, pulled in by hand
  exclude:
    nodes: [default.legacy_revenue]      # deprecated, dropped from the view

Updates & staleness

• Lineage view: resolve live (cheap); no cache, no invalidation.
• Join view: cache since tracing usable joins is expensive. Treat as a derived cache re-traced from the live graph.
• Invalidation is eventual -- since this is for discovery/exploration, strong consistency isn't required. Refresh on a batch cadence (ideally the same cadence as node indexing) plus an explicit "refresh" action.

[ChenyuanZ]

In addition to description, how about add a metadata field to it so clients can supply ai_context?

[shangyian]

@ChenyuanZ yeah, that's a good idea, and I think client-supplied context is worth having, but it's not really the focus of this issue. Collections already exist today, so adding a metadata field is pretty orthogonal to the subgraph rules, which are what this issue is about.

It's also net-new on collections (they don't carry metadata today). Do you want to open a follow-up issue for that?