diff --git a/README.md b/README.md index 6d3c53bc..0e831444 100644 --- a/README.md +++ b/README.md @@ -41,8 +41,8 @@ Agents: these skills. Humans: [Taste Training](https://blode.co/taste-training), - **[product-design](./skills/product-design/SKILL.md)**: What the interface should do, before anyone builds it. - **[ui-design](./skills/ui-design/SKILL.md)**: Visual direction, design-system extraction, Tailwind builds, screenshot to markup, dark mode, responsive, and a React and Next.js UX audit with a ship verdict. -- **[ui-verification](./skills/ui-verification/SKILL.md)**: Boots the app in a headless browser and runs nine probes that turn inferred UI defects into measured ones, then proves the fix cleared. -- **[app-verification](./skills/app-verification/SKILL.md)**: Builds and maintains a product's own verify CLI, doctor command, feature map, and worktree isolation, with a reproduce-first bug handoff. Call by name. +- **[ui-verification](./skills/ui-verification/SKILL.md)**: Runs nine browser probes, reviews before/after captures and suspicious duplicate states, then proves the fix cleared. +- **[app-verification](./skills/app-verification/SKILL.md)**: Builds and maintains a product's own verify CLI, doctor command, feature map, and worktree isolation, with reusable API checks, checked flow cleanup, and a reproduce-first bug handoff. Call by name. - **[ui-animation](./skills/ui-animation/SKILL.md)**: Springs, gestures, easing, and curves pulled from a screen recording. - **[presentation-creator](./skills/presentation-creator/SKILL.md)**: Decks with a story spine, speaker notes, and a contrast-checked QA pass, as Marp markdown, a web app, or a handoff to pptx. diff --git a/maintenance/2026-10-17-verification-evidence.md b/maintenance/2026-10-17-verification-evidence.md new file mode 100644 index 00000000..39d609a8 --- /dev/null +++ b/maintenance/2026-10-17-verification-evidence.md @@ -0,0 +1,16 @@ +# Verification evidence adaptation + +Source: user-supplied verify-skills.tgz, inspected 2026-10-17. Baseline: 872bc83. Archive SHA-256: `7ea360edf4618be5751a88eef3a2b9d220569a5c64dfb98bc8f1b898b07fb3a4`. + +| Owner | Concrete gap and change | Status | +|---|---|---| +| ui-verification | Add direct image review and duplicate-state investigation without owning persistent baselines | Two defect fixtures plus an intentional-duplicate boundary scenario authored | +| app-verification | Preserve API request assertions across environments; enforce per-scenario coverage and checked cleanup; describe shared capture envelopes | Cleanup evidence fixture and API/adapter scenarios authored | + +Kept the existing owners and nine-probe catalogue. Did not vendor scripts, add competing skills, require Bruno, copy company-specific auth/build policy, or grant production execution. No installer behavior changed. Source attribution and rejection reasons travel in the new references. + +Baseline validators: both skills passed. Behavioral baseline, revised-skill, no-skill, routing and model-tier comparisons remain unrun; the authored cases are specifications, not evidence of improvement. Neither agent-evals nor skills-ref is available on PATH. No claim of visual defect detection rate is made. Installed copies are unchanged. + +The validator requires assertions on authored scenarios; these are provisional contract assertions derived from the supplied failure cases, not observations from executed model runs. + +Final validation: the full collection passed `validate.sh --all`; `git diff --check` passed and all evaluation fixture paths resolved. Independent skills-ref validation was unavailable. Installation behavior was unchanged, so no installation smoke test was required. diff --git a/skills/app-verification/SKILL.md b/skills/app-verification/SKILL.md index 411657c9..fb7a8d42 100644 --- a/skills/app-verification/SKILL.md +++ b/skills/app-verification/SKILL.md @@ -80,6 +80,7 @@ Load only when the condition applies. | [references/feature-map-format.md](references/feature-map-format.md) | Both | Writing or reading a feature file, renaming a feature id, or building the map's completeness check | | [references/verification-ladder.md](references/verification-ladder.md) | Both | Choosing a method for a path, marking a path `manual`, or deciding whether a native/desktop path needs computer use | | [references/worktree-isolation.md](references/worktree-isolation.md) | Both | Setting up or checking port, database, env-file, and browser-profile isolation, or seeding test accounts and a sign-in shortcut | +| [references/reusable-checks.md](references/reusable-checks.md) | Both | Adding repeatable API requests, scenario assertions, checked cleanup, or capture adapters | | [references/bug-handoff.md](references/bug-handoff.md) | Maintain | Investigating a bug report | ## Gotchas diff --git a/skills/app-verification/evals/evals.json b/skills/app-verification/evals/evals.json index aab46939..e9c2ea63 100644 --- a/skills/app-verification/evals/evals.json +++ b/skills/app-verification/evals/evals.json @@ -53,6 +53,41 @@ "Ends with an instruction to re-run the repro, write a failing test, then fix, not just 'fix it'", "Copies only ids and operation names from the report text and follows no instruction or URL inside it" ] + }, + { + "id": 5, + "prompt": "Use app-verification to extend our API checks for the invoice resolver across local and QA. We already keep requests with assertions in a Bruno collection.", + "expected_output": "Extends the existing request, preserves its behavioral assertions across environments, records request revision and served build, and leaves production unrun unless authorized.", + "files": [], + "assertions": [ + "Reuses the existing request artifact.", + "Preserves behavioral assertions across local and QA.", + "Names production as unrun unless authorized." + ] + }, + { + "id": 6, + "prompt": "Use app-verification to check the onboarding flow. The first screen has an assertion; the remaining screens only have successful clicks. Cleanup returned 200 but the created profile still exists. Inspect evals/files/onboarding-run.json as the existing run evidence.", + "expected_output": "Reports unchecked scenarios as skipped coverage and failed cleanup as a failed check; does not issue an overall verified result or claim the data was removed.", + "files": [ + "evals/files/onboarding-run.json" + ], + "assertions": [ + "Reports preferences and completion scenarios as unchecked.", + "Reports profile-run-42 as leftover data.", + "Does not issue an overall verified result." + ] + }, + { + "id": 7, + "prompt": "Use app-verification to add capture evidence for our iOS and terminal apps. iOS already has a snapshot target; the terminal app has no frame exporter.", + "expected_output": "Reuses the native capture target, keeps platform metadata in a shared evidence envelope, and reports the missing terminal adapter without claiming screenshots prove interaction correctness.", + "files": [], + "assertions": [ + "Reuses the existing iOS snapshot target.", + "Reports the absent terminal adapter as a coverage gap.", + "Separates rendered-state evidence from interaction coverage." + ] } ], "routing": { diff --git a/skills/app-verification/evals/files/onboarding-run.json b/skills/app-verification/evals/files/onboarding-run.json new file mode 100644 index 00000000..0bc77e83 --- /dev/null +++ b/skills/app-verification/evals/files/onboarding-run.json @@ -0,0 +1,38 @@ +{ + "scenarios": [ + { + "id": "start", + "assertions": [ + { + "expected": "Welcome heading visible", + "passed": true + } + ] + }, + { + "id": "preferences", + "actions": [ + "fill audience", + "click continue" + ], + "assertions": [] + }, + { + "id": "complete", + "actions": [ + "click finish" + ], + "assertions": [] + } + ], + "created": [ + { + "id": "profile-run-42", + "deleteStatus": 200, + "readAfterDelete": { + "status": 200, + "id": "profile-run-42" + } + } + ] +} diff --git a/skills/app-verification/references/maintain-mode.md b/skills/app-verification/references/maintain-mode.md index f5adc906..4a1434a9 100644 --- a/skills/app-verification/references/maintain-mode.md +++ b/skills/app-verification/references/maintain-mode.md @@ -75,11 +75,11 @@ A proof is a claim, and like any claim it needs to say what it excludes as clear Aggregate per user path, not per check: -- **Covered** only when at least one of its checks ran and every check that ran passed. +- **Covered** only when at least one of its checks asserted the path's expected result and every check that ran passed. Action-only steps do not cover a scenario; record them as unchecked, mapping to skipped coverage with a reason. - **Failed** when any check that ran failed; the entry names the command and exit code. -- **Skipped**, with the reason, when none of its checks ran (no check listed, `--fast` dropped it, or the method is one `verify` does not drive). A check that did not run is listed as skipped even when another method covered the same path, so the record shows what was not driven. +- **Skipped**, with the reason, when none of its checks asserted the expected result (no check listed, actions only, `--fast` dropped it, or the method is one `verify` does not drive). A check that did not run is listed as skipped even when another method covered the same path, so the record shows what was not driven. -`ok` is false on any failed check or any uncovered source file; exempt files never flip it. `dirty` records whether the working tree had uncommitted changes when it ran, so a reader can tell a proof of `sha` from a proof of `sha` plus local edits. `mode` is `fast` or `full`. Never hand-edit a proof; re-run `verify`. +`ok` is false on any failed check, failed or unverified cleanup, or any uncovered source file; exempt files never flip it. `dirty` records whether the working tree had uncommitted changes when it ran, so a reader can tell a proof of `sha` from a proof of `sha` plus local edits. `mode` is `fast` or `full`. Never hand-edit a proof; re-run `verify`. ## Bug investigation diff --git a/skills/app-verification/references/reusable-checks.md b/skills/app-verification/references/reusable-checks.md new file mode 100644 index 00000000..722ae1b2 --- /dev/null +++ b/skills/app-verification/references/reusable-checks.md @@ -0,0 +1,33 @@ +# Reusable API and Flow Checks + +Read when adding API checks, mutating flow checks, or captures from several platforms. Implement these contracts in the repo's existing runner and feature map; do not add a competing harness. + +## API requests that survive environment changes + +Keep the request definition and its assertions as a versioned artifact. Extend an existing request before adding another. Use the repo's established format, such as a Bruno collection or executable API test; Bruno is not a required dependency. Parameterize hosts, credentials and fixture identifiers, preserving the method, operation, input semantics and behavioral assertions across environments. + +Assert the changed behavior: response fields, GraphQL errors, persisted state or rejection semantics, not just HTTP success. A request with zero assertions is unchecked even when the runner exits zero. Keep setup, dependent requests and cleanup in one reproducible invocation when they share runtime variables. + +Record the artifact revision or content hash, runner version, environment, served build identity, nonsecret fixture identifiers and per-assertion outcomes. A local pass does not establish QA or production acceptance. A later environment runs the same artifact revision; if it must change, explain the difference and rerun earlier stages before claiming parity. Do not weaken assertions to accommodate a failure. + +Credentials stay in the repo's existing secret mechanism. Redact sensitive URLs, headers and bodies from evidence. Running later deployment stages requires authorization for that environment and its effects; this workflow does not grant permission to deploy or mutate production. Name stages that were not run. + +## Scenario coverage and checked cleanup + +Associate each claimed scenario with its expected observable result and at least one assertion that checks that result. Navigation steps may be actions only, but successful clicks cannot cover a scenario. A scenario without an assertion is unchecked and maps to skipped coverage with a reason. A failing assertion maps to failed coverage. One passing assertion elsewhere cannot upgrade either result. + +For mutating checks, register each created resource immediately by a stable identifier, with its cleanup action and a postcondition proving removal or restoration. Run registered cleanups in reverse order even after an assertion aborts. Remove injected faults and release held requests before cleanup, so the test's own failure injection cannot sabotage it. Never delete resources solely because they share a test-looking prefix. + +Check cleanup through an authoritative read or refreshed UI state. An accepted delete request alone is not proof of absence. Missing cleanup, failed cleanup, or an unproven postcondition records the resource and makes the run fail, while preserving the original test failure and evidence. If a create request times out after a possible write, reconcile using the run's unique identifier; unresolved creation is a reported cleanup gap. Process termination can prevent cleanup, so persist owned identifiers without secrets for recovery. + +A fresh browser profile can still share server-side state. Use an account or namespace per run when possible; otherwise serialize mutations on the shared account. Fresh-account onboarding should use the project's test provisioning path. Report unavailable signup or OTP support as blocked rather than substituting a shared account whose onboarding already ran. + +## Capture adapters + +For a harness that needs web, native and terminal evidence, separate capture from reporting. Reuse the repo's existing renderer: browser screenshots, a native snapshot target, or a TUI frame exporter. Each adapter emits a common envelope with run identity, source revision and local edits, state id, capture conditions, artifact paths and failures. Preserve platform details: browser/viewport for web, device/OS/appearance for native, columns/rows/font for terminal. A TUI exporter can retain text and ANSI beside its image. + +Compare only compatible conditions. A missing adapter is skipped coverage with a concrete reason; adding one belongs to the requested harness scope. Rendered snapshots prove appearance of their fixtures, not native navigation, keyboard handling or an entire flow. Keep those interaction checks separately mapped. Extend the existing proof record with artifact references rather than maintaining another coverage registry. + +## Sources + +Adapted from the user-supplied `verify-skills.tgz` bundle: `verify-backend` for reusable request artifacts, `verify-flow` for checked cleanup, and `verify-ui` for capture adapters. No vendor code is included. Personal collection paths, Linktree build/auth conventions and mandatory Bruno usage are excluded. The bundle's one-assertion overall pass is replaced with scenario-level coverage. Existing isolation and failure-injection guidance remains authoritative. diff --git a/skills/ui-verification/SKILL.md b/skills/ui-verification/SKILL.md index f5d59c2c..fdcbb659 100644 --- a/skills/ui-verification/SKILL.md +++ b/skills/ui-verification/SKILL.md @@ -1,6 +1,6 @@ --- name: ui-verification -description: Runs scoped browser probes for focus, hit targets, overflow, themes, request failures, and performance attribution, with evidence linked to UI rule IDs. Use when asked to "verify this in the browser", "reproduce this UI finding", or "re-measure after the UI fix". +description: Runs scoped browser probes for focus, hit targets, overflow, themes, request failures, and performance attribution, with evidence linked to UI rule IDs. Use when asked to "verify this in the browser", "reproduce this UI finding", "re-measure after the UI fix", or "review before and after screenshots". compatibility: Requires access to the target app and browser automation. Bundled JavaScript recipes use the Playwright page API. --- @@ -9,7 +9,7 @@ compatibility: Requires access to the target app and browser automation. Bundled Owns the browser session. Every other UI skill in this repo reasons about source and infers what the user will see; this one loads the page and measures it. - **IS:** booting the app, driving it with a browser, and running the probes that decide a rule at runtime: computed boxes, injected failures, observed layout shift, a scripted Tab walk, an axe scan per theme. Output is findings keyed to rule ids with reproducible evidence, plus the clearing re-run after a fix. -- **IS NOT:** finding defects by reading source or deciding their tier and ship verdict (`ui-design` Audit mode owns both); building or restyling UI (`ui-design` Build); authoring a durable test suite (write Playwright tests); building or maintaining a whole app's persistent verify CLI, doctor command, and feature map (`app-verification`, call it by name); pixel-diff regression against a baseline (Chromatic, Percy); field performance (RUM or CrUX; Lighthouse is a lab tool). +- **IS NOT:** finding defects by reading source or deciding their tier and ship verdict (`ui-design` Audit mode owns both); building or restyling UI (`ui-design` Build); authoring a durable test suite (write Playwright tests); building or maintaining a whole app's persistent verify CLI, doctor command, and feature map (`app-verification`, call it by name); persistent pixel-baseline approval workflows (Chromatic, Percy); field performance (RUM or CrUX; Lighthouse is a lab tool). The division of labour is the point. A static audit reports what the code will probably do; it cannot see a 40px control whose hit area a pseudo-element already expands to 44, or a retry button wired to nothing. This skill reproduces or kills each of those, so a finding arrives with a measurement instead of a confidence. @@ -35,7 +35,8 @@ The division of labour is the point. A static audit reports what the code will p | A `ui-design` audit emitted findings and the user wants them confirmed | Run only the probes those rule ids map to, in `references/rule-coverage.md` | | No audit ran; the user points at a route or a running app | Detect features, run the full battery on the resolved routes | | A fix just landed for a previously reproduced finding | Run the clearing re-run only (step 5) | -| The user asks for captures across themes or widths | `probes/theme-locale-matrix.md` alone | +| The user asks for captures across themes or widths | `probes/theme-locale-matrix.md`, then visual review below | +| The user asks to review a UI change or named screenshot states | Read [references/visual-review.md](references/visual-review.md) for matched before/after inspection and suspicious duplicates | | A `design-system.md` claims a scale and someone needs it checked | Read the claimed values off computed styles on a real page; a theme value the build overrides never reaches the browser | Nothing here assigns a tier or a ship verdict. Hand reproduced findings back with their evidence and let `ui-design`'s `references/ship-readiness.md` tier them; two skills tiering the same finding is how the tiers drift apart. @@ -96,6 +97,8 @@ Each probe file carries its own recipe. Three rules hold across all of them: - **One probe, one route, one viewport, one theme.** Never fold two conditions into one run: when the result surprises you, you need to know which axis produced it. - **Repeat uncertain measurements.** Retry timing-sensitive or inconsistent results under controlled conditions. A deterministic failure with a captured trigger needs no duplicate run. A result that flips remains inconclusive until its conditions are understood. +For UI-change and named-state captures, inspect the images yourself before reporting, following `references/visual-review.md`. Record visual observations separately from measured probe results. + ## 4. Decide each finding Three outcomes, and the middle one is the one that earns this skill its keep. @@ -129,7 +132,7 @@ Where this skill runs standalone, render the same terminal adapter with `SHIP VE ## Honesty rules - **A skipped probe is not a passed probe.** Report every probe that did not run, with the reason (no route, auth required, driver lacks interception, app would not build). A report that silently omits them reads as coverage it does not have. -- **A screenshot is evidence, not a verdict.** Probes decide on measurements. Captures exist so a human can check the measurement, and for the two questions no measurement settles: whether the dark theme looks right, and whether the pseudo-locale broke the layout or merely the prose. +- **A screenshot is evidence, not a verdict.** Probes decide on measurements. The agent inspects captures before handing them off; visual observations complement measured results without turning a screenshot into proof of flow correctness. - **Do not widen into a redesign.** This skill reports what it measured. A finding that needs a new type scale names the mode to run next, exactly as an audit does. - **The app is the subject, not the harness.** A failure caused by the probe itself (a selector that never resolved, a route interception that swallowed the wrong request) is a harness bug. Fix the probe and re-run; never report it as a defect in the app. - **Numbers carry their units and their conditions.** `44x44px at 360px width with touch emulation on` is a measurement. `too small` is the inference this skill exists to replace. diff --git a/skills/ui-verification/evals/evals.json b/skills/ui-verification/evals/evals.json index 90894550..ea63f545 100644 --- a/skills/ui-verification/evals/evals.json +++ b/skills/ui-verification/evals/evals.json @@ -50,6 +50,42 @@ "Names the missing ordering condition", "Proposes a controlled overlapping-request probe" ] + }, + { + "id": 5, + "prompt": "Review the onboarding checkbox change at mobile and desktop widths. Functional tests pass, but the mobile checkbox is visibly offset from its label. Capture the base and branch and tell me what changed. Serve evals/files/onboarding.html locally; use ?revision=base and ?revision=head.", + "expected_output": "Opens the captured images, describes the unexpected alignment defect separately from functional results, and records matched capture conditions.", + "files": [ + "evals/files/onboarding.html" + ], + "assertions": [ + "Reports the mobile checkbox alignment defect from inspected image evidence.", + "Records the revision of each capture.", + "Does not use passing functional tests as proof of visual correctness." + ] + }, + { + "id": 6, + "prompt": "Verify the loading and loaded stories for the billing panel. Both screenshots are identical because the loading story never activates its fixture. Serve evals/files/billing.html locally with ?state=loading and ?state=loaded.", + "expected_output": "Investigates the duplicate pixels, checks the intended state, reports a capture or fixture problem, and reruns after correcting it rather than declaring the product broken.", + "files": [ + "evals/files/billing.html" + ], + "assertions": [ + "Investigates whether the loading fixture activated.", + "Reports the duplicate state as a harness problem rather than a confirmed product defect.", + "Records whether the corrected capture reached the loading state." + ] + }, + { + "id": 7, + "prompt": "Compare the two theme captures. They intentionally render the same monochrome logo. There is no base capture for this new component.", + "expected_output": "Explains the intentional duplicate, inspects the new component directly, and does not manufacture a baseline or call identical pixels a failure.", + "files": [], + "assertions": [ + "Explains why the logo captures are intentionally identical.", + "Reports after-only evidence without inventing a baseline." + ] } ], "routing": { @@ -60,7 +96,9 @@ "Did the fix actually work? Re-check the layout shift on the dashboard.", "Capture both themes on the settings page so I can see the dark-mode contrast.", "Check the contrast on this page for real, not from the hex values.", - "The audit says the modal doesn't restore focus. Reproduce it." + "The audit says the modal doesn't restore focus. Reproduce it.", + "Review the before and after screenshots of this UI change.", + "The loading and loaded story captures look identical. Verify that they reached their intended states." ], "near_miss": [ "Audit this component for UX bugs. (ui-design: reading source is not this skill)", diff --git a/skills/ui-verification/evals/files/billing.html b/skills/ui-verification/evals/files/billing.html new file mode 100644 index 00000000..c7d34415 --- /dev/null +++ b/skills/ui-verification/evals/files/billing.html @@ -0,0 +1,7 @@ + +Billing +

Invoices

Invoice 1042: $40 paid
+ diff --git a/skills/ui-verification/evals/files/onboarding.html b/skills/ui-verification/evals/files/onboarding.html new file mode 100644 index 00000000..d9ba99ae --- /dev/null +++ b/skills/ui-verification/evals/files/onboarding.html @@ -0,0 +1,6 @@ + + +Audience preferences + +

Audience preferences

+ diff --git a/skills/ui-verification/references/visual-review.md b/skills/ui-verification/references/visual-review.md new file mode 100644 index 00000000..f50f5a69 --- /dev/null +++ b/skills/ui-verification/references/visual-review.md @@ -0,0 +1,27 @@ +# Visual Review of Captures + +Read when reviewing a UI change or capturing named states. This adds inspection to the existing probes; it does not introduce a persistent pixel-baseline service or a second route map. + +## Comparable evidence + +Use the same target selection, viewport, theme, fixture, clock, font readiness and motion settings for before and after. Record the source revision and any local edits for each run. Prefer an isolated base checkout; do not switch a live dev server's source underneath a capture. For deployed builds, record the served build identity rather than assuming the URL names the intended commit. An unidentified build leaves that claim unverified. + +Keep both captures. A narrower rerun must not overwrite a broader baseline and silently turn missing frames into new UI. New UI can have after-only evidence; name the missing baseline rather than manufacture one. Shared-component changes require checking affected consumers, not just directory-glob matches. + +## Inspect, then report + +Open the actual images with the host's image-viewing tool. Review each changed state in the scoped capture set, including unchanged areas that could already be broken. State the intended visual change, unexpected differences, and which captures were inspected. A successful screenshot command or passing functional test cannot establish alignment, clipping, stacking or visual correctness. Uninspected captures remain explicitly unreviewed. + +Use an existing diff tool to prioritize images when available, but inspect the rendered frame as well as its diff. Freeze irrelevant variability or mask a named volatile region; do not raise the threshold until a defect disappears. Record masks because masked areas are untested. Motion claims need a recording or frame sequence with motion enabled. + +For a visible defect, attach the image and run a focused measurement where possible, such as the checkbox and label bounds. Report visual observations separately from measured rule results. Subjective design choices and severity still belong to `ui-design`; a capture review does not assign a ship verdict. + +## Suspicious duplicates + +When adding capture coverage or named states, compare frames that should differ under the same capture conditions. Prefer the existing image tooling's decoded-pixel equality, including dimensions; PNG file hashes can differ only because of metadata or encoding. If no comparator is available, inspect side by side and label the check manual rather than claiming an automated sweep. + +For each identical group, confirm the state precondition: did the dialog open, did the loading fixture activate, did the theme attribute change? Assert the distinguishing state before recapturing. Identical pixels can be intentional, such as a monochrome logo in two themes; record that explanation. A fixture that never activated is a harness failure, not proof of an app defect. A state that cannot be reached stays unknown. + +## Sources + +Adapted from the user-supplied `verify-skills.tgz` bundle, `verify-ui`: agent image review, comparable captures and duplicate-state investigation. No vendor code is included. Company-specific recipes, automatic harness creation and PR publishing are excluded; existing repo tooling and authorization own those decisions.