fix(datasets): attribute subpackage exports per module, matching tasks/ by ethan-scitix · Pull Request #54 · scitix/sieval

ethan-scitix · 2026-07-30T07:29:22Z

Type

fix — resolve a three-way disagreement on subpackage export attribution, and
pin it with the test that would have caught it

Summary

Three sites resolve subpackage exports for the lazy registries, and they did
not agree:

Site	Recorded module
`sieval/tasks/__init__.py:76` (runtime)	`subpkg.module_stem`
`scripts/sync_package_stubs.py:99` (top-level task stub)	bare `subpkg`
`sieval/datasets/__init__.py` + `sync_package_stubs.py:169`	bare `subpkg`

Rows 1 and 2 are a real divergence within the task side — latent only
because no task subpackage is currently registered. The first one to land
would have put from .subpkg import X in the stub while runtime imports
sieval.tasks.subpkg.<module>.

Settle on subpkg.module_stem everywhere. That is the form the documented
rule requires: sieval/tasks/CLAUDE.md mandates an empty subpackage
__init__.py, and an empty __init__ cannot re-export anything, so the
registry must resolve to the defining module. The dataset side had no
documented subpackage convention, which is how the opposite form arrived with
the first real dataset subpackage (datasets/ruler/, #11) — so this adds one.

Two concrete bugs fixed:

Silent overwrite. The duplicate-export guard only fires when the
recorded module differs, so attributing every module in a subpackage to the
bare subpackage name let two same-named exports collide with no error.
The stub↔runtime attribution gap had no test. test_stub_exports_match_runtime_exports
compares only export names (each side renders those from its own scan's
keys), and preflight's check_tasks/check_datasets build their fqn from
the runtime map alone — so neither ever compares the stub's opinion. A
stub attributing an export to the wrong module shipped silently, and ty
would resolve from sieval.datasets import X through a module that does not
define X.

Why neither package generates per-subpackage `init.pyi`

Only a lazy package has a runtime export map for a stub to mirror. A subpackage is
an ordinary package, so a generated stub describes nothing real — and because a stub
shadows the real __init__.py instead of supplementing it, it becomes the
authoritative type surface while contradicting runtime. On the dataset side that
breaks two independent ways (both reproduced):

datasets/ruler/__init__.py re-exports from the private _shared.py, invisible
to suffix-based discovery. The generated stub lists only the class exports →
len_tag/thinking_prefill become unresolved imports at
tasks/ruler_0shot_gen.py:39.
datasets/downloaders/ is infrastructure, not a benchmark, and none of its API
carries a dataset suffix. The generated stub comes out __all__ = [], hiding
resolve() → breaks cli/_readiness.py and cli/dataset/commands.py.

The task side had the same defect in the opposite direction, so the loop that
generated those stubs is now deleted rather than documented. Under the empty
subpackage __init__.py that sieval/tasks/CLAUDE.md prescribes, the stub promised
names the __init__ never binds — verified with a scratch subpackage,
sieval.tasks.<sub>.XTask type-checks but raises AttributeError while top-level
sieval.tasks.XTask resolves correctly. It was inert (zero task subpackages exist,
so no stub was ever emitted and none needed deleting from disk) but armed for the
first one to land. discover_subpackage_tasks, which had no other caller, went with it.

sieval/tasks/CLAUDE.md now also records why that __init__.py is empty, which
was stated without a reason and so read as an oversight next to the dataset side:
nothing imports a task class by name — the registry resolves it through the
top-level package — so a task subpackage needs no type surface of its own, and
staying empty keeps exactly one import path per task. Datasets differ because tasks
import them directly (sieval.datasets is imported 50× in production, 39 of those
from sieval/tasks/; there is no from sieval.tasks import X anywhere in production).

`datasets/ruler/` re-exports narrowed to three

Since the registry no longer depends on them, the 7 names were audited against real
consumers and 4 dropped:

Dropped	Consumers outside `datasets/ruler/`
`RulerTaskSpec`, `ruler_task`	none at all — not a task, not the CLI, not a test, not a non-Python file
`RulerDataset`	production reads it off the top-level registry; only `tests/unit/datasets/test_ruler.py` took the subpackage path
`tokens_to_generate`	test-only, and the other two ruler test modules already import it from `._shared`

What remains is exactly the three names tasks/ruler_0shot_gen.py:39 imports:
RulerDatasetSample, len_tag, thinking_prefill. Two test imports were pointed
at the defining module, matching what the sibling ruler test modules already do.

This trim is only safe because of the attribution fix above: replaying the old
bare-subpackage attribution against the trimmed tree raises module 'sieval.datasets.ruler' has no attribute 'RulerDataset', so the same trim would
have broken the registry before this PR.

Related Issues

Test Plan

Automated

Lint/format clean (ruff check && ruff format --check)
Type check clean (ty check sieval scripts tests → All checks passed)
Unit tests pass — pdm run pytest tests/unit -m "not stress" →
2662 passed
sync_package_stubs.py --check and sync_meta_index.py --check clean
check_preflight.py → 19 PASS, 0 FAIL/WARN. Meaningful here:
check_datasets now builds sieval.datasets.ruler.ruler from the new map
and getattrs the class, so the dotted fqn is exercised end-to-end.

Manual

Attribution is consistent after the change. Stub emits
from .ruler.ruler import (...); runtime map reads
{'RulerDataset': 'ruler.ruler', 'RulerDatasetSample': 'ruler.ruler'};
sieval.datasets.RulerDataset / RulerDatasetSample both resolve.
New test is discriminating (mutation). Repointing one stub import
line while leaving the exported name set untouched fails only the new
test, with test_stub_exports_match_runtime_exports still passing. Verified
both for a flat module (.aime_2024 → .aime_2025) and for the subpackage
(.ruler → .ruler.ruler) — the latter is exactly the half-migrated state
this PR could have produced.
Silent-overwrite hole is really closed. Added a second module under
datasets/ruler/ exporting RulerDataset: now raises
Duplicate dataset export 'RulerDataset' found in 'ruler.ruler' and 'ruler.ztmp_dup'. Replaying the old bare-subpackage logic on the same tree
overwrites silently. Temp module removed.
Subpackage-stub removal is inert. sync_package_stubs.py --check still
passes and both top-level stubs are byte-identical; find sieval -name __init__.pyi returns only the two top-level files, before and after.
Duplicate-export diagnostics now match runtime. Two modules in one
scratch subpackage exporting the same name: generator and runtime both report
found in 'ztmpsub.a_0shot_gen' and 'ztmpsub.b_0shot_gen' (the generator used
to print bare stems). discover_subpackage_tasks' bare-stem contract was
preserved until its removal.

Checklist

Required (all PRs)

PR title follows conventional format (type(scope): description)
No internal paths, credentials, or personal info in committed files
AI-generated code has AI-Generated Code - <model> (<provider>) in
module docstring — extends existing marked files, markers unchanged
No new upper-layer dependencies added to core/
Deleted code verified — discover_subpackage_tasks and the per-subpackage
stub loop are deleted; grepped for dangling references (none in sieval/,
scripts/, tests/, docs/, .pre-commit-config.yaml) and confirmed no
subpackage __init__.pyi existed on disk to clean up. The four dropped
datasets/ruler/ re-exports were each checked for consumers first.

If: Breaking Change

Not breaking for users: _EXPORT_TO_MODULE and the generated .pyi are
internal to lazy loading. Public access (from sieval.datasets import RulerDataset) is unchanged, and both packages' __all__ are byte-identical
before and after.
Existing tests updated to reflect new behavior — only import paths:
tests/unit/datasets/test_ruler.py now takes RulerDataset from
.ruler.ruler and tokens_to_generate from ._shared. No assertion changed;
2662 passed throughout, which is the point of the new test.

🤖 Generated with Claude Code

The generated `.pyi` and the runtime lazy registry both come from their own AST scan of the package (the stub generator must not import the package it generates stubs for), so the two can drift. Two guards already exist and both miss the same half: - `test_stub_exports_match_runtime_exports` compares only export *names*, which each side renders from its own scan's keys. - preflight's `check_tasks` / `check_datasets` build their fqn from the runtime map alone, so they never consult the stub. A stub that attributes an export to the wrong module therefore ships silently, and ty resolves `from sieval.datasets import X` through a module that does not define `X` while runtime resolution stays correct. Compare the stub's per-name `from .mod import X` targets against `_EXPORT_TO_MODULE`. Verified discriminating: repointing one stub import line at a sibling module — leaving the name set untouched — fails only the new test, while the existing `__all__` test still passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two registries resolve subpackage exports, and all three sites disagreed: - `sieval/tasks/__init__.py` (runtime) records "subpkg.module_stem". - `sync_package_stubs.py:99` (top-level task stub) recorded the bare subpackage name — a real divergence from the line above, latent only because no task subpackage is currently registered. The first one would have emitted `from .subpkg import X` while runtime imports `sieval.tasks.subpkg.<module>`. - `sieval/datasets/__init__.py` + `sync_package_stubs.py:169` agreed with each other, but on the bare-subpackage form — the opposite convention. Settle on "subpkg.module_stem" everywhere, which is the one the documented rule requires: `sieval/tasks/CLAUDE.md` mandates an *empty* subpackage `__init__.py`, and an empty `__init__` cannot re-export anything, so the registry has to resolve to the defining module. The dataset side had no documented subpackage convention at all, which is how the divergence got in with the first real dataset subpackage; add one. This also closes a silent-overwrite hole. The duplicate-export guard only fires when the recorded module differs, so attributing every module in a subpackage to the bare subpackage name made two same-named exports collide without error. Verified: a second module in `datasets/ruler/` exporting `RulerDataset` now raises "Duplicate dataset export 'RulerDataset' found in 'ruler.ruler' and 'ruler.ztmp_dup'"; under the old form it overwrote silently. `sieval/datasets/ruler/__init__.py` keeps its re-exports — `tasks/ ruler_0shot_gen.py` imports `len_tag`/`thinking_prefill` from the package, and those carry no registry-visible suffix. What changes is that the registry no longer depends on them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Task subpackages get their own `__init__.pyi`; dataset subpackages do not, and that asymmetry looked like an oversight. It is load-bearing — a generated stub *shadows* the real `__init__.py` instead of supplementing it, so adding the symmetric loop breaks type checking two independent ways (both reproduced): - `datasets/ruler/__init__.py` re-exports 5 of its 7 names from the private `_shared.py`, which suffix-based discovery cannot see. The generated stub lists only the 2 class exports, so `len_tag`/`thinking_prefill` become unresolved imports at `tasks/ruler_0shot_gen.py:39`. - `datasets/downloaders/` is infrastructure, not a benchmark, and none of its API carries a dataset suffix. The generated stub comes out `__all__ = []`, hiding `resolve()` and breaking `cli/_readiness.py` + `cli/dataset/commands.py`. `_iter_subpackage_dirs` cannot distinguish either case from a benchmark subpackage, so the rule is documented rather than automated. Also note the task-side loop is not a model to copy: under the empty- `__init__.py` rule in `sieval/tasks/CLAUDE.md`, the stub it writes promises names the empty `__init__` never binds — verified with a scratch subpackage, `sieval.tasks.<sub>.XTask` type-checks but raises AttributeError, while the top-level `sieval.tasks.XTask` resolves correctly. Left in place rather than deleted: it is inert today, but benchmark task subpackages are expected back, and their `__init__.py` contents are not visible from this tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rationale belongs in the commit messages and PR body, which already carry it. Keep only what a reader cannot infer from the code, stated once: 72 lines of comment/docstring down to 26. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…errors `_discover_task_classes` / `_discover_dataset_classes` guarded duplicates using bare module stems, so a same-subpackage collision read "found in 'a_0shot_gen' and 'b_0shot_gen'" while the runtime registry reported the qualified "ztmpsub.a_0shot_gen". Take an optional `prefix` and build the qualified name inside the helper instead of re-wrapping it at the two call sites: the messages now match the runtime byte-for-byte, and the qualified name is produced in one place rather than three. Behaviour-preserving — `--check` confirms the generated stubs are byte-identical, and `discover_subpackage_tasks` still passes no prefix, so per-subpackage stubs keep their bare stems. Also restore the caveat the previous trim cut from the task-subpackage loop: `sieval/tasks/CLAUDE.md` mandates an empty subpackage `__init__.py`, so that stub promises names the `__init__` never binds — `sieval.tasks.<sub>.XTask` type-checks but raises AttributeError. That is not inferable from the code and is a trap for the first task subpackage to land. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The trim cut the one clause explaining why `scripts/sync_package_stubs.py` carries a second copy of this AST scan: the stub generator must not import the package it generates stubs for. What replaced it ("must agree") states the coupling but not the constraint, leaving two copies of the same scanner — including a duplicated suffix tuple — reading as an obvious dedup target. Move it back into `_discover_dataset_exports`'s docstring and drop the now redundant trailing sentence from the subpackage comment, so it is still stated once. Also fix `_read_stub_import_map`'s docstring: it maps the names the stub *imports*, not the ones it exports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The note said `sieval/tasks/CLAUDE.md` *mandates* an empty subpackage `__init__.py`. The source does not carry that weight: "with an empty `__init__.py`" is a subordinate clause in a bullet about the ≥ 5-file threshold, there is no MUST and no prohibition on re-exporting, and nothing enforces it — no preflight check, no test, and zero task subpackages currently exist to demonstrate it. The datasets side does the opposite, which this PR just documented as correct there. State it as the convention it is, and say what actually follows: the generated stub is wrong for the documented default layout and correct only for a subpackage that re-exports. That is the sharper version of the caveat — the stub silently requires deviating from the convention in order to be accurate. Cite the file without a line number; a pinned line into a markdown file drifts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Only a lazy package has a runtime export map for a stub to mirror. A subpackage is an ordinary package, so the generated `__init__.pyi` described nothing real — and because a stub *shadows* the real `__init__.py` instead of supplementing it, it became the authoritative type surface while contradicting runtime: with the empty subpackage `__init__.py` that `sieval/tasks/CLAUDE.md` prescribes, `sieval.tasks.<sub>.XTask` type-checks but raises AttributeError. This is the same failure the dataset side already avoids by not generating them, so drop the loop rather than keep documenting the trap. Deleting it also removes the caveat comment added earlier in this branch, and leaves `discover_subpackage_tasks` with no callers. Latent, not live: zero task subpackages exist, so no stub was ever emitted and none needed deleting from disk. Verified `--check` still passes and the two top-level stubs are unchanged. Also record *why* a task subpackage's `__init__.py` is empty, which was stated without a reason and so read as an oversight next to the dataset side: nothing imports a task class by name (the registry resolves it through the top-level package), so the subpackage needs no type surface, and staying empty keeps exactly one import path per task. Datasets differ because tasks import them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`datasets/CLAUDE.md` says to re-export only what other packages import directly. `RulerTaskSpec` and `ruler_task` are imported by nothing outside `datasets/ruler/` — not by a task, not by the CLI, not by a test, and not from a non-Python file. Inside the subpackage every user already reaches them via `from ._shared import ...`, which is unaffected. Neither carries a registry-visible suffix, so the lazy export map and both generated stubs are unchanged (`--check` confirms). Also sort `__all__`, which had `tokens_to_generate` ahead of `thinking_prefill`. Left in place for now, though also unused by production: `RulerDataset` (only `tests/unit/datasets/test_ruler.py` takes the subpackage path; production reads it off the top-level registry) and `tokens_to_generate` (test-only, and the other two ruler test modules already import it from `._shared` instead). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…onsumer What is left after the previous commit is still wider than the rule in `datasets/CLAUDE.md` allows. Neither remaining extra serves production: - `RulerDataset` is read off the top-level registry everywhere; only `tests/unit/datasets/test_ruler.py` took the subpackage path. Keeping it also offered a second import path beside `sieval.datasets.RulerDataset`. - `tokens_to_generate` is test-only, and the other two ruler test modules already import it from `._shared` — this file was the odd one out. Point both at the defining module, which is what the sibling test modules do (`test_ruler_0shot_gen.py` imports from `._shared` and `.ruler.ruler`) and what the tests carve-out in the root CLAUDE.md permits. `__init__.py` is now exactly the three names `tasks/ruler_0shot_gen.py` imports. This trim is only safe because of the attribution fix earlier in this branch: the registry resolves `RulerDataset` through `ruler.ruler`, so it no longer depends on the package re-exporting it. Verified the counterfactual — replaying the old bare-subpackage attribution against this tree raises "module 'sieval.datasets.ruler' has no attribute 'RulerDataset'", so the same trim would have broken the registry before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ethan-scitix force-pushed the fix/stub-module-attribution-test branch from 373ff41 to 6818306 Compare July 30, 2026 07:34

ethan-scitix changed the title ~~test(lazy-exports): pin the stub's module attribution to the runtime map~~ fix(datasets): attribute subpackage exports per module, matching tasks/ Jul 30, 2026

ethan-scitix and others added 8 commits July 30, 2026 15:52

ethan-scitix merged commit f14a8f8 into main Jul 30, 2026
9 checks passed

ethan-scitix deleted the fix/stub-module-attribution-test branch July 30, 2026 12:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(datasets): attribute subpackage exports per module, matching tasks/ - #54

fix(datasets): attribute subpackage exports per module, matching tasks/#54
ethan-scitix merged 10 commits into
mainfrom
fix/stub-module-attribution-test

ethan-scitix commented Jul 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ethan-scitix commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type

Summary

Why neither package generates per-subpackage __init__.pyi

datasets/ruler/ re-exports narrowed to three

Related Issues

Test Plan

Automated

Manual

Checklist

Required (all PRs)

If: Breaking Change

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ethan-scitix commented Jul 30, 2026 •

edited

Loading

Why neither package generates per-subpackage `init.pyi`

`datasets/ruler/` re-exports narrowed to three