feat: add otel test conventions and instrumentation sdk#91
Merged
Conversation
Define the OpenTelemetry semantic conventions for tests (span names, resource and case attributes, status mapping) as the canonical machine-readable source in the sdk, and build the instrumentation on top: an L1 fingerprint with os-agnostic path normalization and order-independent params hashing, a TestRunRecorder that accumulates a run and emits a contract-valid ingest batch, an emitRunSpans helper producing real OTel run and case spans with exception events, and a fail-open IngestClient with auth, idempotency headers and offline buffering. Document the conventions and the OTLP to contracts mapping in docs/otel-conventions.md. Closes #12. Closes #13.
AKogut
added a commit
that referenced
this pull request
Jul 17, 2026
PR-2 of M1: the primary ingestion source. A Playwright `Reporter` built on the SDK (#91). ## What's included - **`FlakemetryReporter`** implementing the Playwright `Reporter` interface: `onBegin` → resolve CI run context, `onTestEnd` → record each attempt, `onEnd` → build + deliver the ingest batch - **Status mapping**: `passed@retry0 → pass`, `passed@retry>0 → flaky`, `failed/timedOut/interrupted → fail`, `skipped → skip` - **Retry linkage**: attempts of the same test share identity; a retry points at its first attempt's index - **CI context** from GitHub Actions env (provider, commit, branch, run id, pr number from the ref), with local fallbacks - **Fail-open delivery**: no token/endpoint → delivery is a no-op; an upload error never fails the test process - `FLAKEMETRY_OUTPUT_FILE` dumps the batch to disk for inspection - `examples/` with a ready `playwright.config.ts` + README ## Verification - 12 tests (46 across the workspace): status mapping, suite derivation from describe blocks, env → run context incl. pr number, idempotency key strategy, and a **full-lifecycle integration test** driving synthetic Playwright events → a contract-valid batch (4 executions, flaky retry linked to its first attempt via `retryOfIndex`, suites distinguished) - `build` / `lint` / `typecheck` / `test` green, `format:check` clean ## Notes - An in-repo Playwright fixture run collides with vitest in the shared `node_modules` (both register `expect`), so the end-to-end path is exercised via the lifecycle driver with real Playwright types; a real run against the docker stack is covered by #29 - The batch `resource` carries no `project` — per the contract (#4) the ingest token determines the project server-side Closes #14.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First M1 PR: the OpenTelemetry foundation every reporter builds on. Combines the conventions spec (#12) and the SDK that implements it (#13) — they are inseparable.
OTel semantic conventions (#12)
packages/sdk/src/conventions.ts— canonical constants: span names (test.run/test.case/test.step), resource attributes (flakemetry.project,vcs.commit_sha, …), case attributes (test.identity.fingerprint,test.status,test.attempt, …),CONVENTIONS_VERSIONdocs/otel-conventions.md— human-readable spec: hierarchy, attribute tables, status mapping, fingerprint definition, OTLP → contracts field mappingSDK instrumentation (#13)
sha256(normalized_path + suite + title + params_hash); path normalized workspace-relative / POSIX / lowercase so macOS vs Linux CI don't fork identity; params hashed order-independentlyTestRunRecorder— accumulates a run, computes identity per test, produces a contract-validingestRunBatchemitRunSpans— real OTel spans via@opentelemetry/api: a run span parenting one case span per attempt, exception events on failures, verdict → span statusIngestClient— POST with bearer auth + idempotency header, fail-open (never throws into the test process), optional offline disk bufferingVerification
InMemorySpanExporterincl. exception events, client headers + ack parsing, fail-open on network errorbuild/lint/typecheck/testgreen,format:checkcleanDesign note
The SDK deliberately supports both outputs — OTel spans and the
ingestRunBatchpayload — so the reporter (#14) and api (#15) pick the path, while the server-side identity engine (#18) resolves moves/renames on top of this L1 fingerprint.Closes #12. Closes #13.