Skip to content

1221, 1222: Speed up get_pairwise_comparisons() and skip unused tests in add_relative_skill() - #1227

Open
seabbs-bot wants to merge 4 commits into
mainfrom
claude/hub-work-blocker-vwp2q9
Open

1221, 1222: Speed up get_pairwise_comparisons() and skip unused tests in add_relative_skill()#1227
seabbs-bot wants to merge 4 commits into
mainfrom
claude/hub-work-blocker-vwp2q9

Conversation

@seabbs-bot

@seabbs-bot seabbs-bot commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR closes #1221 and closes #1222.

Pivot scores once (#1221)

pairwise_comparison_one_group() previously re-merged the scores of two comparators for every pair of comparators.
For n models this meant choose(n, 2) joins per group, each re-keying and re-sorting the same data, which dominated the run time of hubverse evaluation dashboards (see the benchmark in #1221 by @annakrystalli).

This PR follows the approach proposed in the issue:

  • Scores are pivoted once per group into a forecast unit by comparator matrix using a new internal helper .pivot_scores().
    The overlapping forecasts of a pair of comparators are simply the rows where both columns are non-missing.
  • The per-pair statistics (mean score ratio, Wilcoxon or permutation p-value) are moved into a shared internal helper .compare_scores().
  • compare_forecasts() is retained as a simple per-pair reference implementation and now delegates to .compare_scores().
    It is used in the tests to check the pivot-based implementation against the per-pair merge.
  • Scores with more than one row per forecast unit and comparator now produce an informative error instead of silently comparing duplicated rows (the old code included such rows via a cartesian merge).
    Exact duplicate rows are still removed, as before.

Results are identical to the previous implementation.
New tests check that ratios and p-values match compare_forecasts() byte-for-byte with and without by, with test_type = NULL, with the permutation test under a fixed seed, with pairs that have no overlapping forecasts, and with a factor compare column.

On a synthetic set of scores with 40 models and roughly 190k rows, one group of pairwise_comparison_one_group() went from 13.7 s and 9.2 GB allocated to 5.2 s and 1.3 GB allocated, with identical output.

Skip the discarded test in add_relative_skill() (#1222)

add_relative_skill() drops the pval and adj_pval columns before returning but previously still ran a Wilcoxon test for every pair of comparators.
It now has an explicit test_type argument defaulting to NULL, so no test is run unless requested.
Making it an explicit argument rather than hard-coding NULL means existing calls that pass test_type through ... keep working.
Relative skill scores are unchanged, and spurious wilcox.test() warnings for tied scores no longer appear.
A test mocks wilcox.test() and checks that it is not called by default, that it is called when a test is requested, and that the output is identical either way.

Checklist

  • My PR is based on a package issue and I have explicitly linked it.
  • I have included the target issue or issues in the PR title as follows: issue-number: PR title
  • I have tested my changes locally.
  • I have added or updated unit tests where necessary.
  • I have updated the documentation if required.
  • I have built the package locally and run rebuilt docs using roxygen2.
  • My code follows the established coding standards and I have run lintr::lint_package() to check for style issues introduced by my changes.
  • I have added a news item linked to this PR.
  • I have reviewed CI checks for this PR and addressed them as far as I am able.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17

Pivot the scores of each group into a forecast unit by comparator matrix
once, instead of merging the scores separately for every pair of
comparators. The overlapping forecasts of a pair are the rows where both
columns are non-missing. Results are identical to the previous
implementation.

The per-pair statistics are moved into a new internal helper,
.compare_scores(), which is shared with compare_forecasts(). The latter
is retained as a reference implementation and is used in tests to check
the pivot-based implementation against the per-pair merge.

Scores with more than one row per forecast unit and comparator now
produce an informative error instead of silently comparing duplicated
rows.

Closes #1221

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.41%. Comparing base (ee3934b) to head (2019494).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1227      +/-   ##
==========================================
+ Coverage   98.38%   98.41%   +0.02%     
==========================================
  Files          41       41              
  Lines        2235     2274      +39     
==========================================
+ Hits         2199     2238      +39     
  Misses         36       36              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

add_relative_skill() drops the p-value columns before returning, but
previously still ran a Wilcoxon test for every pair of comparators. It
now has an explicit test_type argument defaulting to NULL so that no
test is run unless requested. Relative skill scores are unchanged and
spurious wilcox.test() warnings for tied scores no longer appear.

Closes #1222

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
@seabbs-bot seabbs-bot changed the title 1221: Speed up get_pairwise_comparisons() by pivoting scores once 1221, 1222: Speed up get_pairwise_comparisons() and skip unused tests in add_relative_skill() Sep 6, 2026
seabbs and others added 2 commits September 6, 2026 08:02
Use a small subset with tied but non-zero paired differences so that
wilcox.test() warns about ties on all supported R versions. Identical
scores for two models no longer trigger the warning on R 4.6.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
Mock wilcox.test() and count its calls instead of relying on its
warnings, whose behaviour with ties differs across R versions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants