Skip to content

Composable footer opening and file pruning - #10049

Draft
joseph-isaacs wants to merge 1 commit into
ji/scan-protocol-driverfrom
ji/scan-protocol-footer
Draft

joseph-isaacs wants to merge 1 commit into
ji/scan-protocol-driverfrom
ji/scan-protocol-footer

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Add FooterOpen and FooterPrune under vortex_file::planning. The first stage reads and validates a footer through the planning IO protocol; the second rejects empty files and files whose statistics prove the predicate cannot match. Surviving files pass to a caller-supplied Next<OpenedFile>.

The stages are separate from existing file-opening and scan implementations and use the file crate's private helpers without widening their visibility. End-to-end tests finish with a diagnostic {start, end} morsel that exists only in test support; this PR does not evaluate file data or switch public scan entry points.

Tests cover cached and large footers, known/unknown size, stable requests, failures, conservative pruning with missing statistics, and footer-only reads. Builds, tests, formatting, and lint checks have not been run.

Third of three draft PRs: traits → simple driver → footer opening/pruning. Based on ji/scan-protocol-driver. Further planners and production scheduling are follow-on work.

Stack order:

  1. Scan protocol traits
  2. Simple scan planning driver
  3. Composable footer opening and file pruning (this PR)

Add separate FooterOpen and FooterPrune stages and a caller-supplied continuation. Exercise real footer IO and conservative file-statistics pruning with a diagnostic range morsel confined to test support. Keep existing file and scan entry points unchanged.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the ji/scan-protocol-footer branch from d704c9c to de8be02 Compare October 6, 2026 15:28
@codspeed

codspeed Bot commented Oct 6, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 10.75%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 221 untouched benchmarks
⏩ 2403 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ bitpack_blocked_compress_avx2 6.8 µs 7.6 µs -10.75%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/scan-protocol-footer (de8be02) with ji/scan-protocol-driver (493b7d9)

Open in CodSpeed

Footnotes

  1. 2403 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant