Summary
In the good-first-issue-sweep Step 3 eval suite, the field that checks SKIP issues are shown
as a count summary is true in all four cases, three of them vacuously. Add a case with
several SKIP issues so the rule has something to fail on.
Background
skip_count_shown_as_summary_only encodes a real presentation rule: SKIP issues must appear as
a count line, not as a list of titles and reasons. But of the four existing cases, only
case-2-mixed has any SKIP issues at all — and it has one. The output-spec says that when
there are no SKIP issues the field is true "vacuously satisfied", so the other three expect
true for free.
That means a model that always answers true passes the whole suite on this field, including
one that helpfully lists every skipped issue with its reason. The project's own guidance in
docs/education/eval-driven-development.md names this pattern under "All your cases expect the
same value".
The fix is a new case whose input has enough SKIP issues that listing them is the tempting
behaviour — five or six, with distinct-looking skip reasons.
Where to look
tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/ — add a
new case-5- directory here..../fixtures/case-2-mixed/report.md — the closest existing input to model yours on; it shows
the classification-list format the step receives..../fixtures/case-2-mixed/expected.json — the eight fields your expected.json needs..../fixtures/output-spec.md — the definition of each field, including the vacuous-true rule.tools/skill-evals/evals/good-first-issue-sweep/README.md — the case table to extend, and the
case count to bump.
Acceptance criteria
Estimated effort
~1-2 hours for someone new to the codebase; most of it is writing a realistic input.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
Summary
In the
good-first-issue-sweepStep 3 eval suite, the field that checks SKIP issues are shownas a count summary is
truein all four cases, three of them vacuously. Add a case withseveral SKIP issues so the rule has something to fail on.
Background
skip_count_shown_as_summary_onlyencodes a real presentation rule: SKIP issues must appear asa count line, not as a list of titles and reasons. But of the four existing cases, only
case-2-mixedhas any SKIP issues at all — and it has one. The output-spec says that whenthere are no SKIP issues the field is
true"vacuously satisfied", so the other three expecttruefor free.That means a model that always answers
truepasses the whole suite on this field, includingone that helpfully lists every skipped issue with its reason. The project's own guidance in
docs/education/eval-driven-development.mdnames this pattern under "All your cases expect thesame value".
The fix is a new case whose input has enough SKIP issues that listing them is the tempting
behaviour — five or six, with distinct-looking skip reasons.
Where to look
tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/— add anew
case-5-directory here..../fixtures/case-2-mixed/report.md— the closest existing input to model yours on; it showsthe classification-list format the step receives.
.../fixtures/case-2-mixed/expected.json— the eight fields yourexpected.jsonneeds..../fixtures/output-spec.md— the definition of each field, including the vacuous-true rule.tools/skill-evals/evals/good-first-issue-sweep/README.md— the case table to extend, and thecase count to bump.
Acceptance criteria
one READY issue so the rest of the output is exercised.
expected.jsonsets all eight fields from the output-spec.PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/good-first-issue-sweep/runs without error.Estimated effort
~1-2 hours for someone new to the codebase; most of it is writing a realistic input.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.