fix(archive): require a real bzip2 stream header, not a stray BZh - #202
Merged
Merged
Conversation
The multi-stream guard scanned the compressed file for a second occurrence of the format's magic. For bzip2 that magic is three bytes, and three bytes recur by chance roughly every 16 MB of compressed entropy — so an ordinary ~16 MB .tar.bz2 was refused about 60% of the time, and a 100 MB one essentially always, with an error blaming the archive for being multi-stream. The feature was effectively broken above ~10 MB. The verifier reproduced it: a 14.5 MB single-stream .tar.bz2 built from this repo hit a stray `BZh` at offset 11714699 in every variant tried. A genuine concatenated stream begins with the whole 10-byte header: `BZh`, a level digit 1-9, then the 48-bit block magic (digits of pi) or the end-of-stream magic (digits of sqrt pi). Concatenated streams are whole files, so the byte alignment holds. Chance match ~2^-72. zstd stops guessing altogether: `StreamingDecoder` decodes exactly one frame, so anything still readable behind it is a second frame — an exact answer via `into_inner`, where the 4-byte magic scan would have false-positived on roughly a fifth of 1 GB archives. The generic `contains_magic_after_start` is gone with them; gzip never needed it (`MultiGzDecoder` handles concatenation) and xz is unsupported. Two new tests plus the two existing true-positive ones, all mutation-checked: restoring the bare-magic match fails the false-positive test while the real-second-stream tests still pass, which is the discrimination that matters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SM3dDrioC5KntqjB6WDqjW
zoza1982
force-pushed
the
fix/archive-magic-false-positive
branch
from
September 9, 2026 21:00
e71888e to
b0327b1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The multi-stream guard stops matching bzip2's bare 3-byte magic, and zstd stops guessing from bytes at all.
Why
The guard exists so a concatenated archive is refused rather than silently mounted with its later members missing — good. But it scanned the compressed file for a second occurrence of the format's magic, and bzip2's magic is three bytes. Three bytes recur by chance roughly every 16 MB of compressed entropy:
.tar.bz2.tar.bz2So the feature was effectively broken above ~10 MB, and the error told the user their archive was multi-stream when it was not. The verifier reproduced it end-to-end: a 14.5 MB single-stream
.tar.bz2built from this repo contained a strayBZhat offset 11714699 in every variant tried, andArchiveVfs::openrefused it withcompressed_tar_multi_stream.zstd's 4-byte magic has the same defect at 1/256 the rate — negligible on small files, roughly a one-in-five false refusal on a 1 GB archive.
Closes #
How
bzip2 — a real stream header, not a magic fragment. A genuine concatenated stream begins with all ten bytes:
BZh, a level digit1–9, then either the 48-bit block magic (digits of π) or the end-of-stream magic (digits of √π). Concatenated streams are whole files, so byte alignment holds and a plain byte scan is exact for this case. Chance match ≈ 2⁻⁷².zstd — ask, don't guess.
StreamingDecoderdecodes exactly one frame, so anything still readable behind it is a second frame.into_inner()and a one-byte probe answer it exactly. No heuristic left.The generic
contains_magic_after_start(andCompression::magic, now unused) go with them: gzip never needed the guard (MultiGzDecoderhandles concatenation properly) and xz is unsupported.The old doc comment called a false positive "extremely rare". For a 3-byte magic that was off by many orders of magnitude, and the new comment says so with the numbers.
Testing
The discrimination is what matters here, so the mutation check covers both directions:
a_bare_bzh_inside_the_compressed_body_is_not_a_second_streamfails, whilea_real_second_bzip2_stream_is_still_foundandbzip2_multi_stream_is_refused_not_silently_truncatedstill pass — i.e. it reproduces exactly the old bugzstd_multi_frame_is_refused_not_silently_truncatedfailsa_real_second_bzip2_stream_is_still_foundcovers both the block-magic and the end-of-stream form (an empty second stream has no block magic at all), so the tightened signature is not accidentally narrow.Screenshots / output
No UI change — affected archives now open instead of erroring.
Checklist
maincargo fmt,cargo clippy -D warnings,cargo test,cargo docpass locallyCHANGELOG.mdupdated under "Unreleased"Risk & rollback
🤖 Generated with Claude Code
https://claude.ai/code/session_01SM3dDrioC5KntqjB6WDqjW