Repository navigation
Let a caller decide, per element, whether to read the value at all - #86
Merged
Merged
Conversation
A caller after seven index tags out of a 200 MB file had two options, and both were the wrong shape. SpecificTags names the tags to keep, but it filters at every depth, so naming a sequence kept the sequence and emptied its items. StopBeforePixels stops at one hardcoded tag. Neither can say "walk this file, hand me the headers, and let me decide" -- which is what indexing a study actually needs. ReadOptions.OnElement is handed each element's header after it is decoded and before its value is read, and returns ElementKeep, ElementSkip or ElementStop. The RawDataElement it gets carries the Tag, the VR in effect, the declared Length and the ValueTell -- but not the value, not having read it being the entire point. The []PathStep alongside it names the enclosing sequences and the item of each, so the decision can be made at any depth: an oversized nested sequence can be stepped over element by element, which SpecificTags cannot express at all. The three element loops -- two over a byte slice, one over a ReaderAt -- each carried their own copy of the keep/stop conditions, twelve checks in all. They now call one decideElement at the same point, so the answer is the same by construction rather than by three copies staying in step. StopBeforePixels and SpecificTags are implemented on top of it, and TestOnElementStopMatchesStopBeforePixels holds a hook stopping at (7FE0,0010) against the option, tag for tag. Two elements are never offered: group 0x0002, which the transfer syntax is read from, and (0008,0005) Specific Character Set, which decides how every text value after it decodes. Skipping either changes what the rest of the file means rather than how much of it is kept. BREAKING: SpecificTags no longer empties the sequences it keeps. It now filters the top-level dataset only, which is where pydicom applies it -- read_sequence_item calls read_dataset without passing specific_tags along (filereader.py) -- so a sequence that survives the filter is parsed whole. (3006,0010) used to come back with one item holding zero of its two elements, a dataset no file could have produced. An OnElement hook returning ElementSkip when len(path) > 0 restores the old behaviour. Skipping is real work, not a filter applied after the fact. A defined-length value or a defined-length SQ declares its own extent, so stepping over it is arithmetic. An undefined-length value declares nothing, so the item stream still has to be walked to find the end -- but walking it is cheap and copying it is the whole cost of the element, and TestOnElementSkipUndefinedLengthDoesNotCopyIt measures the difference in bytes allocated. Diagnostics are deliberately not suppressed for a skipped element: they describe the file, not the subset the caller chose to keep. ElementStop is the exception, because the parse ends there. Closes #72. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #72 — streaming as a first-class capability (#51 §12).
What a caller could not do before
Indexing a study means reading a handful of tags out of every file and
ignoring the rest. godicom had two ways to narrow a read, and neither has
that shape:
SpecificTagsnames the tags to keep, but it filtered at every depth —naming a sequence kept the sequence element and emptied its items.
StopBeforePixelsstops at one hardcoded tag.Neither can say walk this file, hand me the headers, let me decide.
What this adds
The hook is called once per element, after the header is decoded and before
the value is read.
hcarries the Tag, the VR in effect, the declared Lengthand the
ValueTellthe value would start at — but not the value, nothaving read it being the point.
pathnames the enclosing sequences and theitem of each, so the decision can be made at any depth: an oversized nested
sequence can be stepped over element by element, which
SpecificTagscannotexpress at all.
Three returns:
ElementKeep(the zero value, so a hook that falls off the endof its own logic parses the file the way godicom always has),
ElementSkip,ElementStop.API sign-off
Both API decisions were put to the user before any code was written, per
AGENTS.md(以创造接口为耻,以复用现有为荣 / 以模糊执行为耻,以寻求确认为荣):ReadOptions.OnElementreturning a three-valuedElementAction, rather than a booleanstop_whenmirroring pydicom's.Three values subsume both existing options; a boolean cannot express skip.
SpecificTagswhile we arehere. Chosen over a top-level-only hook.
BREAKING:
SpecificTagsno longer empties the sequences it keepsThis came out of a probe, not a hunch.
SpecificTags: []Tag{(3006,0010)}onrtstruct.dcmreturned the sequence with one item holding zero of its twoelements — a dataset no file could have produced.
pydicom does not do this.
read_sequence_itemcallsread_datasetwithoutpassing
specific_tagsalong (filereader.py), so a sequence that survivesthe filter is parsed whole.
SpecificTagsnow filters the top-level datasetonly, matching that.
TestSpecificTagsKeepsSequenceItemspins it.Callers who want the old behaviour pass an
OnElementreturningElementSkipwhenlen(path) > 0.Convergence
The three element loops — two over a byte slice, one over a
ReaderAt— eachcarried their own copy of the keep/stop conditions, twelve checks in all. They
now call one
decideElementat the same point, so the three agree byconstruction rather than by three copies staying in step.
TestOnElementStopMatchesStopBeforePixelsholds a hook stopping at(7FE0,0010)against the option, tag for tag.Details worth a look
ValueTellis in file coordinates. The streaming reader parses asequence out of a chunk it copied, so the offset its loop holds is relative
to that chunk. Reporting it unadjusted handed the caller a number pointing
into the wrong element —
decideElementaddsrc.baseOffset, the same wayreadContext.reportdoes forDiagnostic.Offset.TestOnElementValueTellInsideSequencereads the bytes at the reported offsetand compares them to the value from a full read.
so the end can only be found by walking the item stream — but walking it is
cheap and copying it is the whole cost of the element.
TestOnElementSkipUndefinedLengthDoesNotCopyItmeasures it in bytesallocated; reverting the branch drops the saving from 6128 bytes to 512.
file, not the subset the caller kept.
ElementStopis the exception, becausethe parse ends there. Both directions are pinned
(
TestOnElementSkipStillDiagnoses,TestOnElementStopEndsDiagnostics).pathis the reader's own slice, valid only for the duration of the call.Documented rather than copied, matching
bufio.Scanner.Bytes()— the hookruns once per element and most hooks only read it. (
Diagnostic.Pathdoescopy, because diagnostics are rare.)
(0008,0005)Specific Character Set arenever offered. Skipping either changes what the rest of the file means.
Verification
Every behavioural claim was checked by reverting the code that implements it
and watching the new test fail:
valueStart += rc.baseOffsetValueTell 16 points at "\x00\x00…", want "ABCD1234"len(path) == 0scope guarditem 0 has 0 elements, want 2the saving is 512, want at least 6128TestOnElementSkipWholeFileAgreesWithFullReadis the round-trip that catches askip landing on the wrong byte: for each of CT_small / MR_small / rtstruct, one
read per top-level tag keeping only that tag, and the value has to match the
full read.
TestOnElementSkipOnEveryReadPathruns the same hook throughReadFile,ReadBytes, a*bytes.Readerand a non-seekable reader, becauseeach takes a different loop.
Gates:
go build,go vet,staticcheck -checks=all,gofmt -l(clean),go test -count=1 -p 4 ./..., andGOARCH=386 CGO_ENABLED=0 go test ./...(which caught
MustTag(0xFFFCFFFC)overflowinginton a 32-bit build).Follow-up
docs/godicom/datasets.mdand itszhcounterpart describe the oldSpecificTagssequence behaviour and need a docs-site PR.🤖 Generated with Claude Code