fix(ci): fix codecov PR coverage disparity with symmetric upload flags#626
Open
fix(ci): fix codecov PR coverage disparity with symmetric upload flags#626
Conversation
Daily scheduled tests (staging + production) upload additional coverage to codecov for main's SHA, but PR commits never receive these uploads. This causes PRs to show ~65% coverage vs main's ~76%, failing the 70% project target despite having equivalent code quality. Fix: tag each upload type with a flag and use carryforward=true for scheduled flags so that PR commits inherit the BASE commit's scheduled test coverage when they don't have their own. - ci flag: PR/push CI uploads (carryforward=false, fresh each run) - scheduled_staging: daily staging uploads (carryforward=true) - scheduled_production: daily production uploads (carryforward=true) When a PR CI runs, codecov carries forward scheduled_* from BASE, giving HEAD = ci(PR) ∪ scheduled_*(BASE) ≈ 76% → passes 70% target.
akunft
requested review from
a team and
helmut-hoffer-von-ankershoffen
as code owners
There was a problem hiding this comment.
Pull request overview
This PR adjusts Codecov configuration and GitHub Actions uploads to use Codecov “flags” with carryforward enabled for scheduled daily coverage uploads, so PRs can inherit the scheduled coverage from the base commit and avoid an artificial PR-vs-main coverage disparity.
Changes:
- Define Codecov flags for
ci,scheduled_staging, andscheduled_production, enabling carryforward for scheduled flags. - Tag the standard CI Codecov upload with the
ciflag. - Tag the daily scheduled workflow Codecov upload with
scheduled_staging/scheduled_productionbased on the environment input.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
codecov.yml |
Adds flag definitions and enables carryforward for scheduled coverage flags. |
.github/workflows/_test.yml |
Adds flags: ci to the Codecov upload step for regular CI runs. |
.github/workflows/_scheduled-test-daily.yml |
Adds environment-derived scheduled_* flags to Codecov uploads for daily scheduled runs. |
Root cause: daily scheduled tests (staging + production) upload additional coverage to codecov for main's SHA, giving main 3 merged uploads (~76%) while PRs only receive 1 upload (~65%), failing the 70% project target. Carryforward would be wrong here — it propagates stale data into PRs and is designed for monorepos with intentionally partial test runs, not for the asymmetric-upload problem described in Codecov's own docs. Correct fix: make PR vs main comparison symmetric by separating upload types at the flag level. - PR CI uploads with NO flag → feeds the "default" project check - Scheduled daily tests upload with flags (staging/production) → informational - Codecov's "default" check sees ONLY unflagged uploads on both sides: ~65% (PR CI) vs ~65% (main's CI upload) → symmetric → target: auto passes target changed from absolute 70% to auto+threshold:2% because: - The 70% was calibrated against 3-upload accumulated coverage, not 1-upload - auto prevents coverage regression between commits - threshold:2% allows small variance without blocking PRs
akunft
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes PRs falsely failing the `codecov/project` check (~65% vs ~76% on main).
Root Cause
Daily scheduled tests (staging + production) upload extra coverage for main's SHA, giving main 3 merged uploads → ~76%. PRs get 1 upload → ~65%, failing the 70% absolute target.
Correct Fix: Symmetric Uploads via Flags
Codecov's `default` project check only counts unflagged uploads when flags are present. Fix:
Both PR and main now have 1 unflagged upload each (~65%), making comparison symmetric.
Coverage Enforcement Unchanged
Enforcement is not relaxed — it is equally strict or stricter:
`target: auto` is stricter than a fixed floor: if main's CI-only coverage is 67%, a PR cannot drop below 65% — whereas the old 70% floor would have rejected the PR entirely for something main already accepts.
Why Not Carryforward
Carryforward is for monorepos that intentionally run a subset of tests per commit. It introduces stale data risk: if a PR touches code that scheduled tests cover, carried-forward data no longer reflects the PR's actual changes. The asymmetric upload count is the root cause Codecov's own docs identify for this failure mode.
Changes
Test plan