Repository navigation
Conversation
…dling find_file_properties_recursively extracts archive members to temporary files with no size limit: a compressed-expansion bomb (tiny zip, huge decompressed output) streams to disk without bound, exhausting storage during scanning. The 7z path already degrades gracefully with a BytesIOFactory size limit; the zip and tar paths had neither the bound nor the error handling. - Stream zip/tar member extraction in chunks up to MAX_MEMBER_EXTRACTION_SIZE, raising ResourceExhaustionError past the cap (declared archive sizes lie, so the bound is enforced while streaming, not from metadata) - Degrade gracefully on corrupt members (BadZipFile/TarError), matching the 7z path, instead of crashing the scan - Only skip zip entries whose type bits name a non-regular type: entries written by zipfile.writestr carry permission bits without file-type bits (mode 0o600, S_ISREG false) and were previously never scanned - Remove a stray debug print from check_if_legacy_format
Contributor
Author
|
Hi @thomas-chauchefoin-tob, the #325 branch is still focused on bounded archive extraction and consistent error handling, with no changes since the last update. Is there a remaining concern you’d like addressed before review, or should I leave the current scope unchanged? |
Contributor
Author
|
Second bump on this one, sorry — the only review activity so far has been my own first ping on 24 September. The scope has not changed since then, so rather than repeat it: the two things I would most appreciate a maintainer ruling on are
If neither is a concern, I would also rather you close it than leave it hanging — happy to hand the bounded-extraction fix over as-is if the permission part is contentious, or vice versa. Either way, thanks for the time. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
find_file_properties_recursivelyexists to inspect untrusted archives, but its zip and tar paths extracted members to disk with no size limit and no error handling:zipped_file.read(fname)streams the full decompressed output, so a compressed-expansion bomb (tiny archive, huge inflated member) exhausts storage during scanning. The 7z path already handled both concerns (BytesIOFactory(limit=...)+except (OSError, ArchiveError)graceful degradation) — the zip and tar paths matched neither.A second defect in the same loop: entries written by
zipfile.writestrcarry permission bits without file-type bits (external_attr = 0o600 << 16, somode == 0is false andstat.S_ISREG(mode)is false) — those members were silently never scanned.Also removes a stray debug
print("check_if_legacy_format")that fired on every tar-file format check throughidentify_pytorch_file_format→PyTorchModelWrapper.validate_file_format.Fix
MAX_MEMBER_EXTRACTION_SIZE, raisingResourceExhaustionErrorpast the cap — declared archive sizes lie, so the bound is enforced while streaming rather than read from metadata; this aligns the polyglot scanner with the expansion-attack protection the analysis path already hasBadZipFile/TarError/OSError→ member skipped), matching the 7z pathS_IFMT), so permission-only entries (writestrstyle) are scannedValidation
pytest test/test_polyglot.py| sample size:N=21| key metrics: 4 new tests fail pre-fix (extraction uncapped on a 4 MiB-of-zeros member with a 1 KiB cap; stray print present; writestr members skipped) → post-fix21 passed| result:passpytest test/→153 passedruff@0.16.0 check+format --check: cleanRisk / Compatibility
ResourceExhaustionErrorinstead of filling the disk, and corrupt members degrade instead of crashing — both align zip/tar with the 7z path's existing contract.