Skip to content

perf: review only what a push changed and keep paid batches when one batch fails - #19

Merged
hyochan merged 4 commits into
mainfrom
perf/incremental-pr-reviews
Nov 14, 2026
Merged

hyochan merged 4 commits into
mainfrom
perf/incremental-pr-reviews

Conversation

@hyochan

@hyochan hyochan commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Why

  • Every push re-reviewed the whole PR. On feat: add shared external purchase link flow openiap#612, three reviews of the same 91–114-file PR used 2.0M, 2.7M and 2.8M tokens ($5.28, $7.21 and $7.34). Two of them posted nothing.
  • One unusable batch discarded everything. The third review failed at batch 7 because one model response could not be processed. Every batch already paid for was thrown away.

Changes

Review only what a push could have changed

  • Anchor. A new review action records reviewSince = {headSha, mergeBaseSha, uncoveredPaths} from the last review that finished with a complete coverage record.

    • "Finished" means published, or completed with nothing new.
    • "Complete" means it recorded uncoveredFiles and its uncovered list was not cut short.
    • Older reviews have no such record, so the next push reviews the whole PR.
  • Scope. On the next push, the scope is:

    • the files changed on the head since the anchor (it must be an ancestor);
    • the files changed between the old and new merge base, when a merge of the base branch moved it forward;
    • the files the anchor left unreviewed.

    It is reviewed as one review when it fits 60 files and the single diff budget. The summary says "Reviewed the N files affected since <sha>".

  • When the whole PR is reviewed again:

    • a force push;
    • a retargeted or rewound base;
    • a comparison of 300 or more files;
    • any comparison failure;
    • a merge-base mismatch between the scope and the snapshot.
  • No model call is made when nothing reviewable changed:

    • A push that only touched files outside the PR diff, or only deleted files, completes as no_new_changes without posting.
    • A binary or oversized file the push changed is reported as not reviewed.
  • Publication of a follow-up review re-reads only the files its findings and evidence need.

Keep paid batches when one batch fails

  • These failures leave the group's files reported as not reviewed, and the review continues:
    • a billed provider response that could not be processed;
    • invalid JSON;
    • evidence or finding validation failures;
    • review limits.
  • At most two groups may fail this way. The third failure stops the review.
  • These still stop the whole review: rate limits and other provider HTTP errors, credentials, exhausted credit, ambiguous outcomes, billing, quota and a changed PR.

Review

  • Three independent Opus review rounds ran.
  • Round 1: 2 high, 3 medium.
  • Round 2: 1 medium (a base-branch merge could revert base fixes unseen).
  • Round 3: no high or medium issues.

Tests

  • bun run lint, typecheck, format:check, build and all 1914 tests pass.
  • New tests cover:
    • a 61-file review followed by a one-file push (one single review, an inline comment and the since note);
    • a force push;
    • a push outside the PR diff;
    • merge-base movement, retarget and rewind;
    • comparison failure and size limits;
    • anchor eligibility;
    • group isolation, the two-failure cap and isolation rules.

🤖 Generated with Claude Code

…batch fails

Every push re-reviewed the whole pull request: a one-commit follow-up on a 114-file PR cost about $7 and 2.7M tokens. A push after a finished review now reviews only the PR files changed since that head, in one review, as long as they fit; a force push or a larger change still reviews the whole PR, and a push that changes nothing in the PR diff completes without posting.

One batch whose model output could not be used failed the whole review and discarded every batch already paid for. Such a batch is now reported as not reviewed and the review continues; billing, quota, credentials, a changed PR and ambiguous provider outcomes still stop it.

@gitdogbot gitdogbot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitdog found no supported defects in f1e4b2f9aed5. 4 of 8 changed files were not fully reviewed. This is not a merge approval; tests were not run.

Gitdog

@gitdogbot gitdogbot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitdog found no supported defects in 98d66af55db7. 6 of 9 changed files were not fully reviewed. This is not a merge approval; tests were not run.

Gitdog

@hyochan
hyochan merged commit 9a99fd9 into main Nov 14, 2026
7 checks passed
@hyochan
hyochan deleted the perf/incremental-pr-reviews branch November 14, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant