Skip to content

feat: add playwright reporter emitting ingest batches#92

Merged
AKogut merged 2 commits into
mainfrom
feat/14-playwright-reporter
Jul 17, 2026
Merged

feat: add playwright reporter emitting ingest batches#92
AKogut merged 2 commits into
mainfrom
feat/14-playwright-reporter

Conversation

@AKogut

@AKogut AKogut commented Jul 17, 2026

Copy link
Copy Markdown
Owner

PR-2 of M1: the primary ingestion source. A Playwright Reporter built on the SDK (#91).

What's included

  • FlakemetryReporter implementing the Playwright Reporter interface: onBegin → resolve CI run context, onTestEnd → record each attempt, onEnd → build + deliver the ingest batch
  • Status mapping: passed@retry0 → pass, passed@retry>0 → flaky, failed/timedOut/interrupted → fail, skipped → skip
  • Retry linkage: attempts of the same test share identity; a retry points at its first attempt's index
  • CI context from GitHub Actions env (provider, commit, branch, run id, pr number from the ref), with local fallbacks
  • Fail-open delivery: no token/endpoint → delivery is a no-op; an upload error never fails the test process
  • FLAKEMETRY_OUTPUT_FILE dumps the batch to disk for inspection
  • examples/ with a ready playwright.config.ts + README

Verification

  • 12 tests (46 across the workspace): status mapping, suite derivation from describe blocks, env → run context incl. pr number, idempotency key strategy, and a full-lifecycle integration test driving synthetic Playwright events → a contract-valid batch (4 executions, flaky retry linked to its first attempt via retryOfIndex, suites distinguished)
  • build / lint / typecheck / test green, format:check clean

Notes

Closes #14.

AKogut added 2 commits July 17, 2026 12:41
Implement @flakemetry/playwright-reporter on top of the sdk: onBegin resolves the CI run context from the environment, onTestEnd records each attempt with status mapping (retried pass becomes flaky, timeouts and interruptions become fail) and links retries to their first attempt, and onEnd builds a contract-valid ingest batch and delivers it fail-open. Suite names come from describe blocks only, CI metadata and pr number are read from GitHub Actions env, and an output-file mode dumps the batch for inspection. Covered by pure mapping tests and a full-lifecycle integration test driving synthetic Playwright events.

Closes #14.
The reporter lifecycle test leaked GITHUB_SHA from the CI environment because resolveRunContext used nullish coalescing, which does not fall through on empty strings. Normalize empty env vars to undefined so set-but-empty CI variables fall back correctly, and make the lifecycle test hermetic by stubbing all GitHub env vars.
@AKogut AKogut self-assigned this Jul 17, 2026
@AKogut
AKogut merged commit 5a3c3f2 into main Jul 17, 2026
2 checks passed
@AKogut
AKogut deleted the feat/14-playwright-reporter branch July 17, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@flakemetry/playwright-reporter — emit OTel spans

1 participant