Skip to content

✨ publish: end-to-end handshake test against fake GitHub, Jira and Linear - #119

Merged
nicholasjackson merged 4 commits into
feat/publish-linearfrom
feat/publish-e2e
Nov 9, 2026
Merged

nicholasjackson merged 4 commits into
feat/publish-linearfrom
feat/publish-e2e

Conversation

@clubanderson

@clubanderson clubanderson commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Part of #114 — the acceptance criteria that need a test repository with a live hive attached (children reaching stage/010-actionable, claim + direct-fix kick, Fixes #<child> in the PR, the untrusted-reporter case) cannot be closed from this repo, so this PR does not use Fixes.

Stacked on #118 (Linear), which is stacked on #117 (Jira); it needs all three targets.

What

  • tests/e2e/publish_test.go: builds the binary (go build) and runs spektacular publish against fakes of GitHub, Jira Cloud, Jira Data Center and Linear. On every target it asserts the Spek side of docs/publish-hive-handshake.md: RFC title starts RFC:, RFC marker has no task= and ends the body, every child has a marker with task= + RFC URL, hangs off the RFC (sub-issue / Epic parent or DC Epic Link / Linear parent), clears the clarity gate (≥ 80 chars after the marker and digest lines are stripped, no <placeholders>), carries the configured label (labels: [spek] on every target) and none of hive's suppress labels and never the parent label, depends_on edges are filed as the target's blocked-by relation, the RFC checklist has a line per child, frontmatter records the publication and URLs, git provenance is recorded, a dry run makes zero calls, refusals carry next_action.
  • Idempotence is asserted on the fake's side too: issue + relation counts are unchanged after a re-run; an edited spec is refused as publish_conflict until --update, which updates the RFC alone; a closed RFC (GitHub closed / Jira Done / Linear completed and archived) is refused. The Jira fake runs with a lagging search index throughout, so the re-run is shown to find its issues via the recorded URLs rather than the search.
  • Production: publish.<target>.labels / parent_label naming a hive suppress label (hold, needs-human, needs-decision, needs-spec, needs-direction) is refused (publish_target_unconfigured) before any network call — publish.PublishOptions.CheckLabels, covered by a unit test and an e2e.
  • TestMain stats every .go/go.mod/go.sum in the module so go test ./... cache is invalidated by any cmd change (and the e2e make target uses -count=1). git stays on PATH via a symlinked tool dir (no gh), and the project is a real git repo.
  • GitHub fake extracted into internal/publish/github/githubtest (now with Snapshot()/Calls(), assert not require in handlers); cmd/publish_test.go uses it instead of its own fake.
  • Handshake doc §6 now states what the fakes prove (that Spek sends the calls the documentation asks for) and what they cannot (a real DC build's Epic Link, Linear's HTML-comment round-trip) — those remain caveats on ✨ publish: Jira target (Epic + Stories, Cloud and Data Center) #117/✨ publish: Linear target (issue + sub-issues, blocks relations) #118.

Tests

make lint && make test green locally; make e2e ~4s.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me against the declared base, #118. The GitHub fake extraction preserves the existing target-test assertions, and tests/e2e/publish_test.go exercises the built CLI through publication, native parent/dependency relations, frontmatter recording and an unchanged re-run across all four configurations. The documentation correctly leaves live Hive claim/kick and reporter-trust verification open under #114.

Dagger Build and Test is green for this head. Local execution was blocked by the Hive pod policy; this review does not claim an independent local test pass.

— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0

@nicholasjackson

Copy link
Copy Markdown
Collaborator

@clubanderson

  1. Two checks always pass:
  • The clarity-gate check (publish_test.go:223) measures body length including the marker. The marker alone is about 140 characters, more than the 80-character minimum.
  • The suppress-label loop (:226-229) never runs, because no config in the test sets labels. Production has no hold-label filtering, yet the docs (:153-154) list both as verified. Please fix the checks or remove the claims.
  1. Idempotency is only checked through the binary’s own JSON (:253-257). Please also assert the fakes’ issue and relation counts after the re-run. --update and the conflict/closed refusals are never run end to end.
  2. The fakes are more forgiving than the real APIs:
  • The Jira DC fake accepts parent on Stories (:310-340, see the ✨ publish: Jira target (Epic + Stories, Cloud and Data Center) #117 comment).
  • The Cloud fake supports property JQL.
  • The Linear fake routes on strings.Contains(query, "issues(") and stores descriptions verbatim.
  • All of them are instantly consistent, so “a re-run files nothing” can’t fail here.
    These tests confirm that the client agrees with itself, not that it works against the real services.
  1. Stale cache. The e2e doesn’t import cmd, so after a cmd change, go test ./... can report a cached pass. Add -count=1, or a build-ID input to the test, so the cache is invalidated.
  2. Smaller items:
  • Fake handlers call require (FailNow) on handler goroutines.
  • The exported maps are read without the lock.
  • cmd/publish_test.go:648 still has its own GitHub fake.
  • The Jira dry run doesn’t assert zero calls.
  • The "Skills #2" contains-check is loose.
  • PATH= drops git provenance.

…near

Add tests/e2e, which builds the binary and runs `spektacular publish` against
in-process fakes of GitHub, Jira Cloud, Jira Data Center and Linear, asserting
the Spek side of docs/publish-hive-handshake.md on every target: RFC title and
marker, child markers with task= and the RFC URL, parent and blocked-by
relations, clarity-gate body hygiene, no suppress or parent labels on
children, the RFC checklist, frontmatter record, idempotent re-run and a dry
run with no network call. `make e2e` runs it alone; `go test ./...` (and so
CI) already includes it, with no credentials needed.

The GitHub fake moves from github_test.go into a githubtest package so the
e2e and the target tests share it. The handshake doc gains a section on what
is verified and what still needs a live hive.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Address review on #119:

- Measure the clarity gate on what a reader sees: the marker and digest
  lines are stripped before the 80-char floor is applied.
- Configure a label on every target so the no-suppress-label check runs
  over a real label; refuse hive suppress labels (hold, needs-human,
  needs-decision, needs-spec, needs-direction) in publish.<target>.labels
  and parent_label before any network call (publish.PublishOptions.CheckLabels).
- Idempotence on the fake's side too: issue and relation counts are
  unchanged after a re-run, a conflict is refused until --update, --update
  changes the RFC alone, a closed/archived RFC is refused.
- Jira runs with a lagging search index throughout, so the re-run is shown
  to find its issues through the recorded URLs; dry runs make zero calls.
- Read fakes through Snapshot() (githubtest gains Snapshot/Calls, and no
  longer uses require inside a handler).
- TestMain stats every Go source so the test cache cannot return a stale
  pass after a cmd change; git is kept on PATH via a symlinked tool dir
  and the project is a real git repo so provenance is asserted.
- cmd/publish_test uses githubtest instead of its own publishFake.
- docs §6 states what the fakes prove and what they cannot.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

clubanderson commented Nov 8, 2026 •

Copy link
Copy Markdown
Member Author

Thanks — all addressed in 3e27d9c + 8379708 (rebased onto #118's 0cf22c8; the second commit adds locked Update/LinkList/RelationList mutators on the fakes after CI's -race run caught the closed-RFC step writing fake state unlocked — your "exported maps read without the lock" point, from the other side).

1. Checks that always pass. Both fixed. The clarity gate now measures the body with HTML comments (marker + digest line) stripped, trimmed, against the 80-char floor. Every target's config sets labels: [spek], so the suppress-label loop runs over a real label and also asserts the configured label is present. And production now has the filter: publish.PublishOptions.CheckLabels refuses labels/parent_label naming hold, needs-human, needs-decision, needs-spec or needs-direction as publish_target_unconfigured with a next_action pointing at the config key, before any network call — unit test in internal/publish and TestPublish_SuppressLabelsAreRefusedBeforeFiling end to end. Doc §6 now matches what's asserted.

2. Idempotence only via the binary's JSON. Fixed. After the re-run the test asserts the fake's issue and relation counts (parent + blocked-by) are exactly what they were. It then edits the spec (adds a Non-Goals section, which only the RFC carries), asserts publish_conflict naming rfc (body) with --update in next_action and no count change; runs --update and asserts Updated == [rfc], children unchanged, counts unchanged, and a following run is a no-op; then closes the RFC on the fake (GitHub closed, Jira Done, Linear completed + archived) and asserts the refusal even with --update.

3. Fakes more forgiving than the APIs. The specific ones are fixed in the stack: the DC fake refuses parent on non-sub-task types and serves the field list (#117); no fake has property JQL any more — Cloud Find is a token search (#117); the Linear fake routes on the first GraphQL field by regex and normalises stored descriptions (#118); and the Jira fake runs with SearchLag on for the whole Jira e2e, so "a re-run files nothing" would fail without the recorded-URL hint path. On the broader point I agree and §6 now says so explicitly: the fakes are models of the documented APIs, so they prove Spek sends the documented calls, not that a real DC build or Linear workspace behaves as documented. Those two remain stated caveats on #117/#118 rather than claims.

4. Stale cache. Fixed. TestMain stats every .go/go.mod/go.sum in the module, so go test ./... records them as inputs and invalidates the e2e on any cmd change; the e2e make target also uses -count=1.

5. Smaller items, all fixed: fake handlers use assert and return an error response rather than require/FailNow; test code reads fakes through Snapshot() (added to githubtest alongside Calls()) taken under the lock; cmd/publish_test.go's own GitHub fake is gone, it uses githubtest; Jira (and Linear) dry runs assert Calls() == 0; the checklist check is a regex for ^[-*]? [ ] <ref> — with the ref quoted; and PATH is a temp dir containing a symlink to the real git only, with the project initialised as a git repo, so provenance is asserted (commit <sha> in the footer) while gh still cannot be found.

Green locally (make lint && make test) and in CI.

CI runs the suite with -race: the closed-RFC step wrote fake state from
the test goroutine while the server goroutines were still alive, and the
link/relation lists were read the same way. Each fake gains Update (closes
over the issue under the lock) and a locked copy of its links/relations;
the e2e uses those. gofmt the import block in target.go.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me for the partial scope stated in the body. cmd/publish.go:188 checks the configured labels before Find or Publish; the new refusal tests also assert zero fake-server calls. The shared GitHub fake preserves the existing creation, update and relation assertions, while the binary tests check issue/relation counts independently of the command output on re-runs.

Per #114, the live-hive claim/kick and reporter-trust checks remain outstanding; §6 of the handshake doc makes that boundary explicit. Current-head Dagger build/test checks passed. Local execution was blocked by the hive reviewer policy, so I have not independently run the tests.

— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining correctness issue in the cache fix (medium): tests/e2e/publish_test.go:66 calls dependOnSources() before m.Run() at line 76. Go starts recording filesystem inputs for the test cache inside testing.M.Run (m.before() → StartTestLog), so these source-file stats are never recorded. A change confined to cmd can still reuse the previous e2e pass under ordinary go test ./..., skipping both the binary build and the handshake assertions. Move the dependency scan into a test (after logging starts), and verify that changing a command source invalidates a previously cached e2e result. make e2e already avoids this with -count=1, but the default make test path does not.

The dry-run path returns before target construction, and the new label guard runs before Find; the zero-call assertions match those paths. The scope left open under #114 is explicitly documented. Current CI build-and-test checks pass; local test execution was denied by the hive policy.

— hive: agent=reviewer backend=codex model=gpt-6.1-sol codex=0.167.0

@nicholasjackson

Copy link
Copy Markdown
Collaborator

/approve

@kubestellar-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nicholasjackson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@nicholasjackson
nicholasjackson merged commit 8f6fd66 into main Nov 9, 2026
6 of 7 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/publish-e2e branch November 9, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants