GitHub Actions for harness checks#3
Merged
Merged
Conversation
adamlutz
added a commit
that referenced
this pull request
May 19, 2026
The harness has been silently relying on humans (or me, prompted) to
remember to open PRs. The "if it's not in a file, it doesn't exist"
principle from Lecture 03 says that's a hole. This commit fills it.
AGENTS.md → End Of Session — adds step 6 (open a PR if the branch is
reviewable) and step 7 (paste the PR URL into the session entry).
docs/SESSION.md → adds a new "Opening a PR" section with:
- When to open (branch represents coherent work; not gated on the
feature being fully verified)
- The gh pr create template (simple human-readable title, body with
summary / verification status / to-test-locally / merge order)
- After-opening responsibilities (paste URL, watch first CI run)
- Naming guidance (title in plain English; feature ID lives in the
body, not the title — matches PR #1 "Initial harness setup", PR #2
"Maestro end-to-end verification", PR #3 "GitHub Actions for harness
checks")
claude-progress.md → adds `Opened:` line to the session handoff template
so future sessions record the PR they opened (or note explicitly that
no PR was warranted).
Why this lives on ci-001 and not on its own docs branch: opening a PR
is *what makes the CI gates from this feature useful*. The two work
together — the workflow + the discipline of triggering it. Same scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adamlutz
added a commit
that referenced
this pull request
May 19, 2026
PR #2 ("Maestro end-to-end verification") merged into main, conflicting on claude-progress.md (Next-action + Sessions blocks) and feature_list.json (the 4th feature slot). Resolution: - claude-progress.md — kept this branch's narrative (ci-001 in flight, PR #3 open); appended the e2e merge + scaffolding sessions from main so the full history is preserved. - feature_list.json — kept BOTH features (e2e-001 from main + ci-001 from this branch). Now 5 features total: harness-001 done, ui-001/ui-002 todo, e2e-001 todo, ci-001 in_progress. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two-job workflow at .github/workflows/harness.yml:
static (ubuntu-latest, ~1 min, required)
- npm ci
- typecheck, lint, jest, feature_list.json schema
- scripts/harness-ci-checks.sh — 5 harness-specific invariants:
1. claude-progress.md has a "Next action" block
2. no .only / .skip / xit / debugger in committed source
3. every status=done feature has a non-empty commitSha
4. all required harness files present
5. no .env / .p12 / .jks / .key files committed
e2e-ios (macos-latest, ~10–20 min, advisory)
- install Maestro
- boot iOS sim (scripts/sim-ios.sh)
- expo prebuild + xcodebuild for the iphonesimulator SDK
- install + launch on the booted sim
- run .maestro/flows/home.yaml via scripts/run-maestro.sh
- upload tmp/diagnostics/ as a workflow artifact
The e2e-ios job is `continue-on-error: true` until it has been green
for a stretch; it also only runs on main pushes or PRs labelled `e2e`
(macOS minutes are ~10× ubuntu's). Both gates are gated by
`concurrency:` so superseded PR pushes don't queue up runs.
Adds feature `ci-001` to feature_list.json (status=in_progress; flipped
to done once the workflow is observed running on GitHub). WIP=1
preserved — harness-001 is done; this branch holds the only in_progress
feature.
The e2e-ios job depends on .maestro/* and scripts/run-maestro.sh from
feat/e2e-maestro. Merge order: harness-001-close → e2e-maestro →
ci-harness → main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the "Naming standards" section to docs/HARNESS.md and wires it into
the validators so consistency is enforced mechanically rather than by
discipline alone.
The standard, in one sentence: the feature ID is the load-bearing
identifier across `feature_list.json`, the branch name, and the commit
subject. All three must carry the same ID so a single `git log
--grep='(ui-002)'` returns the full story.
| Artefact | Format
| -------------- | ---------------------------------
| Feature ID | <category>-<NNN> (3-digit zero-padded)
| Branch name | <type>/<feature-id>-<slug>
| Commit subject | <type>(<feature-id>): <subject>
Enforcement:
- scripts/feature-list-check.js — feature id regex tightened to
^[a-z][a-z0-9]*(-[a-z][a-z0-9]*)*-\d{3}$
- scripts/harness-ci-checks.sh — new step 6 validates the current branch
name against the <type>/<feature-id>-<slug> pattern and confirms the
embedded id exists in feature_list.json (skips main / release / hotfix).
- AGENTS.md routing map flags HARNESS.md as the home of naming standards.
Also: claude-progress.md updated for the three branch renames that
follow in subsequent push operations.
These conventions are project-specific, not from the harness course. The
course (Lecture 08) only mandates feature-IDs-as-primitives; carrying
the ID into branches and commits is a project extension to keep the
three forms of state queryable as one system.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #4 split claude-progress.md into PROGRESS.md + DECISIONS.md and deleted the speculative lib/ and __tests__/ directories plus several docs. Update harness-ci-checks.sh so its invariants point at what now exists: - Check 1/6: PROGRESS.md → "## Next steps" (was claude-progress.md → "Next action"). - Check 2/6: scan only app/components/hooks for stray debug markers; lib/ and __tests__/ no longer exist. - Check 4/6: REQUIRED set now AGENTS.md, CLAUDE.md, init.sh, feature_list.json, PROGRESS.md, DECISIONS.md, docs/HARNESS.md, docs/SESSION.md, docs/ARCHITECTURE.md, docs/RN_PLATFORM.md (drops claude-progress.md, docs/VERIFICATION.md, docs/E2E_TESTING.md; adds PROGRESS.md, DECISIONS.md, docs/SESSION.md). Verified locally: scripts/harness-ci-checks.sh exits 0 and npm run verify is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adamlutz
force-pushed
the
feat/ci-001-actions
branch
from
337fb7b to
8c50302
Compare
adamlutz
added
the
e2e
The e2e-ios job's `if:` condition reacts to the `e2e` label, but `pull_request` defaults to `[opened, synchronize, reopened]` — so adding the label after the PR exists wouldn't re-evaluate the gate. Adding `labeled` to the trigger types closes that gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First e2e-ios run failed at `xcrun simctl install booted "$APP_PATH"` with "No such file or directory". Two bugs: 1. APP_PATH was captured relative to ios/ (the build step does `cd ios`), but the install step runs from the repo root. Capture the full absolute path via $PWD. 2. APP_ID was hardcoded as com.anonymous.rnharness via job env, but the actual built .app uses whatever CFBundleIdentifier is in its Info.plist (Expo's default is derived from the scheme name). Read the real bundle ID via PlistBuddy and export it; the env-level value is now just a fallback if the read fails. Also adds an explicit existence check + echoed "Resolved APP_PATH/APP_ID" lines so the next failure (if any) is easier to diagnose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The second run got cancelled at 30m with install + launch only 30s in. Build alone took ~28 min on a cold macos-latest runner (16 min on the prior run — significant variability). With install + Maestro + diagnostics needing another 5–10 min, 30 min is too tight without caching. Raising to 60 min lets us actually observe the Maestro flow run while caching is being designed. Plan to drop back to 30 once CocoaPods + DerivedData caching is in place — the build itself should be 5–10 min warm. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last e2e-ios run (#26133345614) hit the 30 min timeout entirely in the build step. expo prebuild + xcodebuild from a cold runner takes ~30 min because there's no cache and --clean blew away ios/ every time. This commit fixes both. Adds actions/cache@v4 over the four paths that dominate native build time: - ios/ prebuild is idempotent - .expo/ Expo prebuild fingerprint - ~/Library/Caches/CocoaPods pod source/binary cache - ~/Library/Developer/Xcode/DerivedData Xcode incremental build cache Cache key: ios-build-<os>-<hash(package-lock.json, app.json)>, with a restore-keys prefix so a near-miss still recovers most of the build. On a cache hit the build step should drop from ~25–30 min → ~8–12 min, roughly 1/3 the macOS budget per run on the free tier. Drops `--clean` from `expo prebuild` so a restored ios/ is actually reused. If a config plugin change ever leaks a stale ios/, bust the cache by changing app.json (which is in the key) rather than re-adding --clean here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 26134632326 finally reached the Maestro step and failed at command
parsing with 'TypeError: Cannot read property "exp" of undefined'.
Cause: `appId: ${APP_ID:-host.exp.exponent}` uses bash parameter
expansion, but Maestro evaluates `${...}` through its GraalVM JS engine.
JS reads `APP_ID:-host.exp.exponent` as a labelled statement + unary
negation + member access on undefined `host`, so the whole flow aborts
before any step runs.
Fix: keep `appId: ${APP_ID}` in the YAML and move the fallback into
scripts/run-maestro.sh (`export APP_ID="${APP_ID:-host.exp.exponent}"`).
CI runs with APP_ID already set to the built bundle's CFBundleIdentifier,
so this only changes the no-env-set default for local Expo-Go runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 26135477495 got past the parse failure but died at flow start with
"Unable to launch app undefined". Maestro's GraalVM JS engine does not
automatically inherit the shell environment — variables referenced as
\${APP_ID} in YAML are resolved from JS globals, not the OS env. The
correct passthrough is `maestro test -e APP_ID=...`.
Also echoes the resolved APP_ID so the next failure (if any) shows
exactly what was passed in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 26136224883 failed before the build even ran with "hashFiles couldn't
finish within 120 seconds". hashFiles patterns are matched as recursive
globs from GITHUB_WORKSPACE, so `hashFiles('package-lock.json', 'app.json')`
walked node_modules looking for nested matches and timed out.
Compute the digest explicitly with shasum over just the two intended files
and pass it as the cache key via a step output. Same auto-bust semantics
(key changes when either file changes), no traversal of node_modules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 26136489399 finally got Maestro to exercise the live screen for 51s before failing on "Assertion is false: \"Welcome!\" is visible". Root cause: the build was -configuration Debug, which expects a running Metro server to serve the JS bundle. CI has no Metro, so the app launches into a blank/red-screen state and every assertVisible fails. Switch to -configuration Release — react-native-xcode.sh now runs at build time and embeds the JS bundle in the .app, so the app is self-contained on the simulator. Also update the .app find path from Debug-iphonesimulator → Release-iphonesimulator. Trade-off: Release builds optimise (slightly slower than Debug, but with the cache the delta is small) in exchange for a CI-runnable app. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run 26137558247's Release build let Maestro actually exercise the live screen. "Welcome!" and "Step 1: Try it" passed; "Step 2: Explore" failed because iPhone 16 Pro renders Step 2 below the fold and `assertVisible` only checks the current viewport. Use `scrollUntilVisible` for Steps 2 and 3 — this both fixes the assertion and provides incidental proof that ParallaxScrollView is interactive, not just statically rendered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Latest run (#26138282501) failed on `scrollUntilVisible: "Step 2: Explore"`
after 36s. The user-applied scrollUntilVisible approach scrolled the
viewport but still couldn't find the text.
Diagnosis: the Expo scaffold wraps "Step 2: Explore" in:
<Link href="/modal">
<Link.Trigger>
<ThemedText type="subtitle">Step 2: Explore</ThemedText>
</Link.Trigger>
...
`Link.Trigger` from expo-router 6 surfaces the text as part of a
context-menu trigger rather than a plain queryable node — Maestro's
accessibility view can't match the literal text, so neither
`assertVisible` nor `scrollUntilVisible` find it regardless of scroll
position.
The Step 1/2/3 lines are scaffold cruft. Once `ui-001` lands they go
away entirely. Dropping them now removes a false-failure source and
keeps the smoke flow honest. What stays:
- assertVisible "Welcome!" bundle loaded, home rendered
- tab bar "Home" / "Explore" Tabs navigator mounted
- tapOn "Explore" router actually works
- takeScreenshot visual record for the agent
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run #26164917513 launched the app and passed `assertVisible: "Welcome!"` (bundle loads, home renders), then failed on `text: "Home"` for the tab bar label after 31s. Same shape as the Step 2/3 failure — Maestro's accessibility introspection can't find the label text, either because of iOS 26's redesigned tab bar layout or the project's HapticTab wrapper swallowing it. Trimming to the minimum that reliably proves the toolchain works: - launchApp - assertVisible: "Welcome!" bundle loaded, home rendered - takeScreenshot: home visual record The dropped checks (Step N, tab labels, tap on Explore) are nice-to-have but not core smoke value — they will move to `ui-001.yaml` once that feature lands with harness-specific UI worth asserting on, at which point we can also use stable accessibility IDs instead of label text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous runs all showed "Cache not found" on every restore because
actions/cache@v4 only writes its post-step on full-job success, and
every prior run failed at Maestro before that step could run. The most
recent run (#26166345238) finally passed end-to-end and did save the
cache — but any future failure between build and Maestro would throw
the build away again.
Splitting into actions/cache/restore@v4 (top of job) +
actions/cache/save@v4 (with `if: always() && cache-hit != 'true'` at
the end) is the canonical robust pattern. After this commit:
- the very next run will be a CACHE HIT (the prior run saved
ios-build-macOS-2aa375aaad3bfec0) → build should drop from ~20 min
to ~8–12 min
- if a future Maestro change breaks the flow but the build still
succeeds, the cache will save anyway and the iteration after that
benefits
Path list duplicated between restore and save (no YAML anchor support
in GHA). Comment in both blocks notes the constraint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adamlutz
added a commit
that referenced
this pull request
May 20, 2026
The repo had drifted from project-specific notes ("the bring-up is
complete", "next available work is ui-001") toward template-shaped
content (everything in scripts/, docs/, .maestro/ is generic harness
machinery). This commit makes the intent explicit.
README rewrite:
- Opens with a "GitHub template — Use this template" callout so the
github.com button is discoverable.
- New "What's included" section listing harness subsystems, init
phase, baseline gate, end-of-session gate, naming standard,
Maestro, sim helpers, GitHub Actions CI.
- New "Conspicuously not included" section — calls out the absence of
lib/, __tests__/, design system, networking. The harness is about
HOW, not WHAT.
- New "First things to edit" section walking a user through the
setup tasks after cloning (README, app.json, feature_list.json,
PROGRESS.md, DECISIONS.md, package.json).
GitHub template flag set via `gh repo edit --template`. Verified:
$ gh repo view --json isTemplate
{"isTemplate": true}
feature_list.json:
- Flipped harness-002 to done at SHA be68289 (was still in_progress
from PR #4's working state; the merge to main happened but the
bookkeeping commit was missing — fixing here).
- Added docs-001 (this feature) with status in_progress. WIP=1 ✓.
PROGRESS.md:
- Updated to reflect harness-002 done, docs-001 in progress, PR #3
still open. Next steps point at landing PR #3 before this PR so
the README's GitHub Actions reference is accurate on main.
The README mentions .github/workflows/harness.yml which lands when
PR #3 merges — this PR should merge after that one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adamlutz
added a commit
that referenced
this pull request
May 20, 2026
PR #3 merged into main while this branch was open; rebase resolved the feature_list.json conflict (kept ci-001 from main, flipped it to done at c8bb4c9; kept docs-001 from this branch). PROGRESS.md was automatically merged but still referenced PR #3 as "open" — fixing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 20, 2026
adamlutz
added a commit
that referenced
this pull request
May 20, 2026
Per user direction: PROGRESS.md's only durable value (the next-best-step
pointer) is derivable from features/ — pick the highest-priority
in_progress feature, else the first todo. Everything else PROGRESS.md
tried to carry was duplicated elsewhere (status in features/, history
in git log, decisions in DECISIONS.md, per-branch context in PR
descriptions).
WHAT'S REMOVED
- PROGRESS.md (deleted)
- All references across AGENTS.md, CLAUDE.md, README.md, init.sh,
docs/{HARNESS,SESSION,ARCHITECTURE}.md, scripts/check-clean-state.sh,
scripts/harness-ci-checks.sh
WHAT'S NEW: CLOSE-BEFORE-MERGE CONVENTION
The in-flight feature's status is flipped to `done` in the LAST commit
on the branch BEFORE the merge, with `commitSha` set to that commit
(the latest implementation commit). The PR's merge then carries the
closed feature into main as part of its own diff. No follow-up
bookkeeping commit.
This is the harness's answer to the recurring post-merge bookkeeping
gap that bit ci-actions (PR #3), harness-features (PR #6), and
docs-template (PR #7) — three consecutive PRs, same shape, same fix
forgotten every time.
Documented in docs/SESSION.md → "The close-before-merge convention"
and surfaced in AGENTS.md's End-of-session steps. scripts/check-clean-state.sh
check 4 now warns if the in-flight feature is still in_progress
(soft warning — the work may legitimately not be ready to merge).
OTHER CHANGES
- init.sh: 7 steps → 6 (no PROGRESS tail at the end)
- scripts/harness-ci-checks.sh: 6 checks → 5 (dropped PROGRESS-has-Next-steps)
- scripts/check-clean-state.sh: check 4 reshaped to check feature status
- DECISIONS.md: replaced the 'trim PROGRESS.md' entry with the
'remove PROGRESS.md' decision documenting the convention
NOTE ON THIS PR'S OWN CLOSE-BEFORE-MERGE
This commit is the implementation. The close-before-merge flip
happens in the next commit on this branch (referencing this commit's
SHA as commitSha).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a two-job GitHub Actions workflow that runs the harness's gates on every push and PR — the CI equivalent of `./init.sh && npm run verify` plus the simulator E2E flow from PR #2.
Also codifies the project's naming standards (feature ID, branch, commit) and tightens the validators so consistency is enforced mechanically rather than by discipline alone.
Summary
`.github/workflows/harness.yml`
Two jobs, gated by `concurrency:` so superseded PR pushes don't pile up runner minutes.
`static` runs: `npm ci` → `typecheck` → `lint` → `jest` → `feature_list.json` schema → `scripts/harness-ci-checks.sh`.
`e2e-ios` runs: install Maestro → boot iOS sim → `expo prebuild` + `xcodebuild` for the iphonesimulator SDK → install + launch on booted sim → run `.maestro/flows/home.yaml` → upload `tmp/diagnostics/` as a workflow artifact.
`scripts/harness-ci-checks.sh`
Six invariants beyond what tsc / lint / jest catch:
Naming standards (docs/HARNESS.md)
A single `git log --grep='(ui-002)'` now returns the full story for `ui-002` across all three forms of state.
Validators
What this PR itself verifies
Opening this PR triggers the `static` workflow against the branch's own files — that is the verification step that flips `ci-001` to `done`. After observing one green CI run, update `feature_list.json[].commitSha` to the merge commit SHA.
To exercise `e2e-ios`: add the `e2e` label to this PR. Once it has been green for a stretch, drop `continue-on-error: true` to make it required.
Notes
Merge order
Third in the chain. PR #1 (merged) → PR #2 → this PR. The `e2e-ios` job will only have anything to run against once PR #2 lands.
🤖 Generated with Claude Code