Skip to content

Scan protocol traits - #10047

Open
joseph-isaacs wants to merge 3 commits into
developfrom
ji/scan-protocol-traits
Open

joseph-isaacs wants to merge 3 commits into
developfrom
ji/scan-protocol-traits

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Introduce composable scan planning through explicit IO and CPU state machines. Live Planner and Morsel implementations are Send and can move between threads with their owning run. Next<Input> constructs a live successor directly; constructors store inputs, while substantial work belongs in compute().

Requests are published exactly once through compute() returning NeedsIO(batch). state() reports Done, NeedsCompute, or Waiting; the driver tracks outstanding fetches, and waiting without one is a protocol error. IoSlot exposes is_waiting() instead of returning outstanding request batches.

Add the protocol under vortex_scan::planning and byte request/result contracts under vortex_io::request. Byte ranges carry alignment, IoSource supports completion polling with a waker, and IoService creates per-scope sessions. Include a simple blocking ReadAtIoSource adapter that declines optional hints. IO owner identities remain independent of scan scopes. Existing scan execution paths are unchanged.

Tests cover request bookkeeping, alignment, direct successor construction and constructor errors, and the reader adapter's registration, delivery, and cleanup behavior. No local builds, tests, lint, or formatting commands were run for this update.

First of three draft PRs: protocol and byte-source adapter → simple driver → footer opening/pruning. Data planners, IO priority/coalescing, and engine integration remain outside this PR. The later stacked PRs will need to adopt these updated contracts.

Stack order:

  1. Scan protocol traits (this PR)
  2. Simple scan planning driver
  3. Composable footer opening and file pruning

Add explicit planner and morsel state machines, deferred stage composition, and byte IO contracts in separate modules. Keep live execution worker-local and IO owner identities independent of scan scopes. Include protocol and construction tests.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2105 untouched benchmarks
⏩ 503 skipped benchmarks1


Comparing ji/scan-protocol-traits (5855fe8) with develop (8b7a257)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs
joseph-isaacs added this pull request to stack #10050 September 25, 2026 12:50
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Sep 25, 2026
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: f82b9c0

Format the new planning modules with the CI-pinned nightly formatter. This corrects the formatting check and the quoted sign-off on the original commit without rewriting history.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 5, 2026 10:10
Require Send live planners and morsels, publish IO requests once, and
construct successor planners directly. Add aligned byte requests, IO
sessions and completion polling, and the blocking reader adapter.
Update the protocol and adapter tests for these contracts.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go with this and tune as we learn more

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants