Skip to content

fix(archive): require a real bzip2 stream header, not a stray BZh - #202

Merged
zoza1982 merged 1 commit into
mainfrom
fix/archive-magic-false-positive
Sep 9, 2026
Merged

zoza1982 merged 1 commit into
mainfrom
fix/archive-magic-false-positive

Conversation

@zoza1982

@zoza1982 zoza1982 commented Sep 9, 2026

Copy link
Copy Markdown
Owner

What

The multi-stream guard stops matching bzip2's bare 3-byte magic, and zstd stops guessing from bytes at all.

Why

The guard exists so a concatenated archive is refused rather than silently mounted with its later members missing — good. But it scanned the compressed file for a second occurrence of the format's magic, and bzip2's magic is three bytes. Three bytes recur by chance roughly every 16 MB of compressed entropy:

Archive size Chance of a false refusal
~16 MB .tar.bz2 ~60%
~100 MB .tar.bz2 essentially always

So the feature was effectively broken above ~10 MB, and the error told the user their archive was multi-stream when it was not. The verifier reproduced it end-to-end: a 14.5 MB single-stream .tar.bz2 built from this repo contained a stray BZh at offset 11714699 in every variant tried, and ArchiveVfs::open refused it with compressed_tar_multi_stream.

zstd's 4-byte magic has the same defect at 1/256 the rate — negligible on small files, roughly a one-in-five false refusal on a 1 GB archive.

Closes #

How

bzip2 — a real stream header, not a magic fragment. A genuine concatenated stream begins with all ten bytes: BZh, a level digit 19, then either the 48-bit block magic (digits of π) or the end-of-stream magic (digits of √π). Concatenated streams are whole files, so byte alignment holds and a plain byte scan is exact for this case. Chance match ≈ 2⁻⁷².

zstd — ask, don't guess. StreamingDecoder decodes exactly one frame, so anything still readable behind it is a second frame. into_inner() and a one-byte probe answer it exactly. No heuristic left.

The generic contains_magic_after_start (and Compression::magic, now unused) go with them: gzip never needed the guard (MultiGzDecoder handles concatenation properly) and xz is unsupported.

The old doc comment called a false positive "extremely rare". For a 3-byte magic that was off by many orders of magnitude, and the new comment says so with the numbers.

Testing

cargo test -p cairn-backend-archive     # 53 passed (+2)
cargo clippy -p cairn-backend-archive --all-targets --all-features -- -D warnings

The discrimination is what matters here, so the mutation check covers both directions:

Mutant Result
restore the bare 3-byte match a_bare_bzh_inside_the_compressed_body_is_not_a_second_stream fails, while a_real_second_bzip2_stream_is_still_found and bzip2_multi_stream_is_refused_not_silently_truncated still pass — i.e. it reproduces exactly the old bug
zstd never reports multi-frame zstd_multi_frame_is_refused_not_silently_truncated fails

a_real_second_bzip2_stream_is_still_found covers both the block-magic and the end-of-stream form (an empty second stream has no block magic at all), so the tightened signature is not accidentally narrow.

Screenshots / output

No UI change — affected archives now open instead of erroring.

Checklist

  • PR title follows Conventional Commits
  • Branched off main
  • cargo fmt, cargo clippy -D warnings, cargo test, cargo doc pass locally
  • Tests added (2, plus the 2 existing true-positive tests as the other half of the mutation check)
  • Docs updated — the rustdoc now states the real collision rate and why bzip2 needs a scan while zstd does not
  • CHANGELOG.md updated under "Unreleased"
  • Security review — n/a; this loosens a guard, so the true-positive tests are the safety argument
  • No secrets, credentials, or generated artifacts committed

Risk & rollback

  • This makes a refusal less likely, so the failure mode to worry about is the opposite one: a genuine multi-stream bzip2 archive mounting with later members missing. The two true-positive tests are exactly that case and both still pass, and the signature is the format's own documented stream header rather than a heuristic.
  • A bzip2 stream whose header is not byte-aligned would be missed — but concatenated streams are whole files, so that cannot arise from concatenation, which is the only way a multi-stream file is produced.
  • Rollback: revert the single commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SM3dDrioC5KntqjB6WDqjW

The multi-stream guard scanned the compressed file for a second occurrence
of the format's magic. For bzip2 that magic is three bytes, and three bytes
recur by chance roughly every 16 MB of compressed entropy — so an ordinary
~16 MB .tar.bz2 was refused about 60% of the time, and a 100 MB one
essentially always, with an error blaming the archive for being multi-stream.
The feature was effectively broken above ~10 MB. The verifier reproduced it:
a 14.5 MB single-stream .tar.bz2 built from this repo hit a stray `BZh` at
offset 11714699 in every variant tried.

A genuine concatenated stream begins with the whole 10-byte header: `BZh`, a
level digit 1-9, then the 48-bit block magic (digits of pi) or the
end-of-stream magic (digits of sqrt pi). Concatenated streams are whole
files, so the byte alignment holds. Chance match ~2^-72.

zstd stops guessing altogether: `StreamingDecoder` decodes exactly one frame,
so anything still readable behind it is a second frame — an exact answer via
`into_inner`, where the 4-byte magic scan would have false-positived on
roughly a fifth of 1 GB archives.

The generic `contains_magic_after_start` is gone with them; gzip never needed
it (`MultiGzDecoder` handles concatenation) and xz is unsupported.

Two new tests plus the two existing true-positive ones, all mutation-checked:
restoring the bare-magic match fails the false-positive test while the
real-second-stream tests still pass, which is the discrimination that matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SM3dDrioC5KntqjB6WDqjW
@zoza1982
zoza1982 force-pushed the fix/archive-magic-false-positive branch from e71888e to b0327b1 Compare September 9, 2026 21:00
@zoza1982
zoza1982 merged commit c4c9853 into main Sep 9, 2026
10 checks passed
@zoza1982
zoza1982 deleted the fix/archive-magic-false-positive branch September 9, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant