[DRAFT RFC] Platform Coverage Metrics via Manifest Adoption

[GarthDB]

Status: DRAFT RFC. No normative decisions yet.

Relationship to other work

Artifact	Where it lives
Umbrella vision	#714 — Design Data Specification
Distributed repos, manifests, layers	#715 — Distributed Design Data Architecture
Platform manifest spec (normative)	spec/manifest.md
Taxonomy backbone (structured names, precondition for platform-owned formatting)	#806 — Token Taxonomy, Vocabulary, and Formatting
Agent-readable surface (CLI/TUI/MCP)	#973 — Interactive TUI & Token Authoring Wizard
RFC index	docs/rfc-coordination.md

Summary

Two metrics run today, informally: design-token version per platform, and AI-generated cross-platform difference reports. Neither is backed by a released tool, and both are behind the original Apr 3 target for KRs 2D/2E. This RFC is the formal proposal Aaron Brownlee asked for at the direction he set at the May 15 Director Review:

"AI as entry point, but we're making the report by running a deterministic, versioned and released tool. 'Make me a report' doesn't make me confident, that's a little bit of smoke and mirrors."

We propose two phases. Phase 1 is a bridge: turn the coverage scanners already built for the Jul 30 baseline into a deterministic, versioned, scheduled tool, so the Sep 30 POC has real numbers behind it. Phase 2 is the fix that actually lasts: every platform declares its relationship to the foundation using the existing platform manifest spec, and coverage metrics become a query over those manifests instead of a set of scanners we maintain by hand. Phase 1 gets us a number by Sep 30. Phase 2 gets us a number we don't have to keep re-earning every time a platform changes its build.

Problem

No platform declares its relationship to the foundation dataset today. Each one just consumes tokens however it consumes them, and the shape of that consumption is different everywhere: SWC 1st-gen resolves var(--spectrum-x), SWC 2nd-gen compiles a token() call through postcss-token, React Spectrum expands colorScale() by regex into hundreds of literal names, and iOS ships its own asset catalogs. There's no common data source to read coverage from, so we scan compiled output instead, once per platform, using a different detector for each one.

That's what produced the Jul 30 baseline: four hand-tuned scripts, each specific to one platform's build output, run manually against a single checkout. They're not scheduled, not in CI, and not resilient to a platform changing its own tooling. If SWC 2nd-gen changes how it compiles token() calls, our detector for it breaks quietly and the number goes stale without anyone noticing.

That's the actual problem: not that we lack a report, but that we lack a shared way for a platform to say what it depends on from the foundation. Everything downstream, coverage numbers included, is a workaround for that missing declaration.

Design principle

Aaron's ask sets the bar for both phases: AI can be the entry point (asking for a report, phrasing a query), but the number itself has to come from a deterministic, versioned, released tool. No AI in the number generation. The Jul 30 baseline already meets this bar (script-generated, re-run twice, byte-identical), and both phases below keep meeting it.

Proposal, Phase 1: productize the coverage scanners (bridge)

Turn the four platform-specific scanners behind the Jul 30 baseline into one released, versioned tool:

• Same per-platform detection logic (SWC1st var() resolution, SWC2nd token() compiled-output scan, RS colorScale() expansion, iOS asset-catalog scan), moved from scratchpad scripts into a maintained package.
• Scheduled or CI-triggered runs instead of manual, so a platform's build change surfaces as a broken detector, not a silently stale number.
• Same coverage model: canonical join key is the kebab-case token name, and a match counts through the full transitive alias closure (a component token that resolves up through semantic and primitive layers counts as depending on the whole chain, not just the literal name it's called with). This was verified against SWC 2nd-gen's compiled CSS, where the full alias chain ships as real custom properties.

This phase feeds the Sep 30 interim commitment: a POC/example metrics report to leadership generated by a real tool, not written by an AI. It's explicitly a bridge. The detectors stay brittle because they're inferring a relationship the platform never declared.

Proposal, Phase 2: manifest-derived metrics (target)

Once a platform's manifest.md declares its foundationVersion pin, include/exclude queries, typed overrides, and extensions, coverage stops being something we infer from compiled output and becomes something we read directly:

• Foundation-usage ratio: tokens included via manifest queries versus the full foundation set at the pinned version.
• Override count and drift: how many tokens a platform overrides, and how far its pinned foundationVersion trails the latest release.
• Cross-platform comparison: because every platform expresses its relationship to the foundation in the same manifest shape, these numbers are directly comparable without a separate detector per platform.

This is the same manifest shape proposed for starter repos (the adoption on-ramp platforms fork to get onto the cascade). A platform adopting a starter repo gets metrics for free, because the manifest it already needs for adoption is the same manifest the metrics tool reads. Once a platform is on a manifest, its Phase 1 scanner retires.

Metrics methodology

Applies to both phases, since Phase 2 doesn't change what's being measured, only where the input comes from.

• Canonical join key: kebab-case token name.
• Coverage: transitive alias closure, not direct call-site match only. A depth-first search over {token} alias references from each call site.
• Denominator: the live-token set (2,150 at the Jul 30 baseline), excluding tokens that are both deprecated and unused. Deprecated-but-still-used tokens stay in the denominator.
• Candidate metrics beyond raw coverage: ratio of hard-coded to token-mapped attribute values in ProtoPack/React apps; amount of foundation data an implementation uses versus values unique to its own build.

Baseline evidence (Jul 30, first cut)

Platform	Direct call-site matches	Dependency closure	% of live tokens (2,150)
SWC 1st-gen	773	913	42.5%
SWC 2nd-gen	417	551	25.6%
React Spectrum	567	582	27.1%
iOS	59	89	4.1%

Two governance findings came out of the set/mode participation check run alongside this baseline, not from coverage itself:

• Token sets are fully symmetric (every scale token ships desktop and mobile, every color token ships light, dark, and wireframe), so set participation doesn't change any platform's count.
• wireframe is authored on 470 tokens with zero product usage across all four platforms. It's a dead set.
• iOS ships mobile-scale tokens only. That's the weakest-evidenced cell in this table and worth confirming directly with the iOS team before treating it as fact.

Timeline

• Sep 30, 2026: Phase 1 tool produces the POC/example metrics report to leadership.
• Nov 20, 2026: baseline metrics analyzed with platform partners, using whichever manifests have landed by then. Coverage for platforms still on Phase 1 detectors comes from the scanner; coverage for platforms on manifests comes from the manifest read directly.

Open questions

• Migration ordering: does a platform's manifest replace its Phase 1 scanner the moment the manifest lands, or do both run in parallel for a verification period?
• Manifest ownership: who on each platform team owns keeping foundationVersion and include/exclude current?
• Denominator policy: confirm the deprecated-but-used exclusion rule holds as the token set grows.
• Metric thresholds: is there a coverage percentage that should trigger a flag, or is this purely descriptive for now?
• Hosting and cadence for the Phase 1 tool: which CI, how often, who owns alerting on a broken detector?
• iOS mobile-scale-only: confirm with the iOS team whether this is intentional or a gap in what we're measuring.

[GarthDB]

The mechanism this RFC's metrics depend on is now live, so it can move off "blocked on manifest adoption existing."

PR #1353 shipped the manifest cascade over a pinned github source, and the iOS POC (GarthDB/spectrum-ios-design-data) is a real manifest repo exercising it end to end. That gives "manifest adoption" a concrete shape to measure against: a platform's manifest.json, with its include/exclude, overrides, and extensions, is exactly the surface a coverage metric would read.

The tooling that would surface adoption (a validation Action, a repo template, platform-scoped querying via MCP) is tracked under epic spectrum-design-data-h890. Worth revisiting the metric definitions here now that there's a manifest to point them at.