1221, 1222: Speed up get_pairwise_comparisons() and skip unused tests in add_relative_skill() - #1227
Open
seabbs-bot wants to merge 4 commits into
Open
1221, 1222: Speed up get_pairwise_comparisons() and skip unused tests in add_relative_skill()#1227seabbs-bot wants to merge 4 commits into
seabbs-bot wants to merge 4 commits into
Conversation
Pivot the scores of each group into a forecast unit by comparator matrix once, instead of merging the scores separately for every pair of comparators. The overlapping forecasts of a pair are the rows where both columns are non-missing. Results are identical to the previous implementation. The per-pair statistics are moved into a new internal helper, .compare_scores(), which is shared with compare_forecasts(). The latter is retained as a reference implementation and is used in tests to check the pivot-based implementation against the per-pair merge. Scores with more than one row per forecast unit and comparator now produce an informative error instead of silently comparing duplicated rows. Closes #1221 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1227 +/- ##
==========================================
+ Coverage 98.38% 98.41% +0.02%
==========================================
Files 41 41
Lines 2235 2274 +39
==========================================
+ Hits 2199 2238 +39
Misses 36 36 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
add_relative_skill() drops the p-value columns before returning, but previously still ran a Wilcoxon test for every pair of comparators. It now has an explicit test_type argument defaulting to NULL so that no test is run unless requested. Relative skill scores are unchanged and spurious wilcox.test() warnings for tied scores no longer appear. Closes #1222 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
Use a small subset with tied but non-zero paired differences so that wilcox.test() warns about ties on all supported R versions. Identical scores for two models no longer trigger the warning on R 4.6. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
Mock wilcox.test() and count its calls instead of relying on its warnings, whose behaviour with ties differs across R versions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR closes #1221 and closes #1222.
Pivot scores once (#1221)
pairwise_comparison_one_group()previously re-merged the scores of two comparators for every pair of comparators.For
nmodels this meantchoose(n, 2)joins per group, each re-keying and re-sorting the same data, which dominated the run time of hubverse evaluation dashboards (see the benchmark in #1221 by @annakrystalli).This PR follows the approach proposed in the issue:
.pivot_scores().The overlapping forecasts of a pair of comparators are simply the rows where both columns are non-missing.
.compare_scores().compare_forecasts()is retained as a simple per-pair reference implementation and now delegates to.compare_scores().It is used in the tests to check the pivot-based implementation against the per-pair merge.
Exact duplicate rows are still removed, as before.
Results are identical to the previous implementation.
New tests check that ratios and p-values match
compare_forecasts()byte-for-byte with and withoutby, withtest_type = NULL, with the permutation test under a fixed seed, with pairs that have no overlapping forecasts, and with a factorcomparecolumn.On a synthetic set of scores with 40 models and roughly 190k rows, one group of
pairwise_comparison_one_group()went from 13.7 s and 9.2 GB allocated to 5.2 s and 1.3 GB allocated, with identical output.Skip the discarded test in
add_relative_skill()(#1222)add_relative_skill()drops thepvalandadj_pvalcolumns before returning but previously still ran a Wilcoxon test for every pair of comparators.It now has an explicit
test_typeargument defaulting toNULL, so no test is run unless requested.Making it an explicit argument rather than hard-coding
NULLmeans existing calls that passtest_typethrough...keep working.Relative skill scores are unchanged, and spurious
wilcox.test()warnings for tied scores no longer appear.A test mocks
wilcox.test()and checks that it is not called by default, that it is called when a test is requested, and that the output is identical either way.Checklist
lintr::lint_package()to check for style issues introduced by my changes.🤖 Generated with Claude Code
https://claude.ai/code/session_01LpZEQ26Z76K1a3SxFKkR17