Skip to content

Schematize custom_metadata and filter on its fields #2330

Description

@shangyian

Summary

[Relates to: #1352]

Nodes carry a free-form custom_metadata JSON field with no schema enforcement and no way to query by its contents. We should add two capabilities:

  • Schematize: let systems register a JSON Schema for an individual custom_metadata key, validated on write.
  • Filter: let clients filter nodes by custom_metadata fields.

Design decisions

Per-key schemas, keyed (key, node_type?, namespace?): custom_metadata is a shared bag with many independent writers, so the unit of schema ownership must match the unit of write ownership: the top-level key. A single whole-object schema is a non-starter (no single
owner). Resolution picks the most specific active row per key: (key,node_type,namespace) > (key,·,namespace) > (key,node_type,·) > (key,·,·).

Lax, write-time-only validation: Keys with a registered schema are validated on write and unregistered keys are ignored. Registering a
schema never retroactively rejects or mutates stored rows. This keeps existing data working and requires no migration.

Two registration surfaces: Ad-hoc via REST API or durable via repo-backed deploy YAML (namespace-scoped), mirroring how nodes are authored. Globally-scoped keys are intended for cross-cutting/legacy keys but new integrations should claim a namespaced key.

Tiered filtering:

  • [Tier 0] any key, no schema: equality/containment via a single global GIN index on the JSONB column (custom_metadata @> '{"k":"v"}'). Works for every key with no per-key setup.
  • [Tier 1] schema registered: typed operators (numeric range/sort) backed by a per-key expression index, plus appearance in a curated, type-validated facet catalog.

Schemas are not a hard gate on basic filtering but they provide better guarantees around quality / discoverability.

Sub-issues

  • Migrate custom_metadata to JSONB + global GIN index
  • Add a per-key custom_metadata schema registry (table + resolution)
  • Validate custom_metadata against registered schemas on write (lax)
  • Filter nodes by custom_metadata fields (DB layer)
  • Registry REST API: CRUD, facet catalog, violations, per-key indexes
  • Expose customMetadataFilters on GraphQL findNodes
  • Register custom_metadata schemas from repo-backed deploys
  • Seed schemas for existing high-volume keys + document the model
  • Follow-ups (pagination counts, dedup, index naming, CONCURRENTLY)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions