Repository navigation
✨ publish: end-to-end handshake test against fake GitHub, Jira and Linear - #119
Conversation
There was a problem hiding this comment.
Looks correct to me against the declared base, #118. The GitHub fake extraction preserves the existing target-test assertions, and tests/e2e/publish_test.go exercises the built CLI through publication, native parent/dependency relations, frontmatter recording and an unchanged re-run across all four configurations. The documentation correctly leaves live Hive claim/kick and reporter-trust verification open under #114.
Dagger Build and Test is green for this head. Local execution was blocked by the Hive pod policy; this review does not claim an independent local test pass.
— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0
|
…near Add tests/e2e, which builds the binary and runs `spektacular publish` against in-process fakes of GitHub, Jira Cloud, Jira Data Center and Linear, asserting the Spek side of docs/publish-hive-handshake.md on every target: RFC title and marker, child markers with task= and the RFC URL, parent and blocked-by relations, clarity-gate body hygiene, no suppress or parent labels on children, the RFC checklist, frontmatter record, idempotent re-run and a dry run with no network call. `make e2e` runs it alone; `go test ./...` (and so CI) already includes it, with no credentials needed. The GitHub fake moves from github_test.go into a githubtest package so the e2e and the target tests share it. The handshake doc gains a section on what is verified and what still needs a live hive. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Address review on #119: - Measure the clarity gate on what a reader sees: the marker and digest lines are stripped before the 80-char floor is applied. - Configure a label on every target so the no-suppress-label check runs over a real label; refuse hive suppress labels (hold, needs-human, needs-decision, needs-spec, needs-direction) in publish.<target>.labels and parent_label before any network call (publish.PublishOptions.CheckLabels). - Idempotence on the fake's side too: issue and relation counts are unchanged after a re-run, a conflict is refused until --update, --update changes the RFC alone, a closed/archived RFC is refused. - Jira runs with a lagging search index throughout, so the re-run is shown to find its issues through the recorded URLs; dry runs make zero calls. - Read fakes through Snapshot() (githubtest gains Snapshot/Calls, and no longer uses require inside a handler). - TestMain stats every Go source so the test cache cannot return a stale pass after a cmd change; git is kept on PATH via a symlinked tool dir and the project is a real git repo so provenance is asserted. - cmd/publish_test uses githubtest instead of its own publishFake. - docs §6 states what the fakes prove and what they cannot. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
74afa64 to
3e27d9c
Compare
|
Thanks — all addressed in 3e27d9c + 8379708 (rebased onto #118's 0cf22c8; the second commit adds locked 1. Checks that always pass. Both fixed. The clarity gate now measures the body with HTML comments (marker + digest line) stripped, trimmed, against the 80-char floor. Every target's config sets 2. Idempotence only via the binary's JSON. Fixed. After the re-run the test asserts the fake's issue and relation counts (parent + blocked-by) are exactly what they were. It then edits the spec (adds a Non-Goals section, which only the RFC carries), asserts 3. Fakes more forgiving than the APIs. The specific ones are fixed in the stack: the DC fake refuses 4. Stale cache. Fixed. 5. Smaller items, all fixed: fake handlers use Green locally ( |
CI runs the suite with -race: the closed-RFC step wrote fake state from the test goroutine while the server goroutines were still alive, and the link/relation lists were read the same way. Each fake gains Update (closes over the issue under the lock) and a locked copy of its links/relations; the e2e uses those. gofmt the import block in target.go. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Looks correct to me for the partial scope stated in the body. cmd/publish.go:188 checks the configured labels before Find or Publish; the new refusal tests also assert zero fake-server calls. The shared GitHub fake preserves the existing creation, update and relation assertions, while the binary tests check issue/relation counts independently of the command output on re-runs.
Per #114, the live-hive claim/kick and reporter-trust checks remain outstanding; §6 of the handshake doc makes that boundary explicit. Current-head Dagger build/test checks passed. Local execution was blocked by the hive reviewer policy, so I have not independently run the tests.
— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
One remaining correctness issue in the cache fix (medium): tests/e2e/publish_test.go:66 calls dependOnSources() before m.Run() at line 76. Go starts recording filesystem inputs for the test cache inside testing.M.Run (m.before() → StartTestLog), so these source-file stats are never recorded. A change confined to cmd can still reuse the previous e2e pass under ordinary go test ./..., skipping both the binary build and the handshake assertions. Move the dependency scan into a test (after logging starts), and verify that changing a command source invalidates a previously cached e2e result. make e2e already avoids this with -count=1, but the default make test path does not.
The dry-run path returns before target construction, and the new label guard runs before Find; the zero-call assertions match those paths. The scope left open under #114 is explicitly documented. Current CI build-and-test checks pass; local test execution was denied by the hive policy.
— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: nicholasjackson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Part of #114 — the acceptance criteria that need a test repository with a live hive attached (children reaching
stage/010-actionable, claim + direct-fix kick,Fixes #<child>in the PR, the untrusted-reporter case) cannot be closed from this repo, so this PR does not useFixes.Stacked on #118 (Linear), which is stacked on #117 (Jira); it needs all three targets.
What
tests/e2e/publish_test.go: builds the binary (go build) and runsspektacular publishagainst fakes of GitHub, Jira Cloud, Jira Data Center and Linear. On every target it asserts the Spek side ofdocs/publish-hive-handshake.md: RFC title startsRFC:, RFC marker has notask=and ends the body, every child has a marker withtask=+ RFC URL, hangs off the RFC (sub-issue / Epic parent or DC Epic Link / Linear parent), clears the clarity gate (≥ 80 chars after the marker and digest lines are stripped, no<placeholders>), carries the configured label (labels: [spek]on every target) and none of hive's suppress labels and never the parent label,depends_onedges are filed as the target's blocked-by relation, the RFC checklist has a line per child, frontmatter records the publication and URLs, git provenance is recorded, a dry run makes zero calls, refusals carrynext_action.publish_conflictuntil--update, which updates the RFC alone; a closed RFC (GitHub closed / Jira Done / Linear completed and archived) is refused. The Jira fake runs with a lagging search index throughout, so the re-run is shown to find its issues via the recorded URLs rather than the search.publish.<target>.labels/parent_labelnaming a hive suppress label (hold,needs-human,needs-decision,needs-spec,needs-direction) is refused (publish_target_unconfigured) before any network call —publish.PublishOptions.CheckLabels, covered by a unit test and an e2e.TestMainstats every.go/go.mod/go.sumin the module sogo test ./...cache is invalidated by any cmd change (and thee2emake target uses-count=1). git stays on PATH via a symlinked tool dir (nogh), and the project is a real git repo.internal/publish/github/githubtest(now withSnapshot()/Calls(),assertnotrequirein handlers);cmd/publish_test.gouses it instead of its own fake.Tests
make lint && make testgreen locally;make e2e~4s.