Skip to content

Implement the four missing pickle opcodes (FLOAT, BYTEARRAY8, NEXT_BUFFER, READONLY_BUFFER) - #324

Open
feiiiiii5 wants to merge 1 commit into
trailofbits:masterfrom
feiiiiii5:fix/implement-missing-opcodes
Open

feiiiiii5 wants to merge 1 commit into
trailofbits:masterfrom
feiiiiii5:fix/implement-missing-opcodes

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Root cause

Fickling's OPCODES_BY_NAME registry covered 64 of the 68 standard pickle opcodes. The four missing ones raise NotImplementedError from Opcode.__new__ at load time — before any analysis runs — so Pickled.load() crashes with a traceback instead of producing a verdict on perfectly benign pickles:

opcode protocol real-world trigger
FLOAT 0 any float literal in a legacy protocol-0 pickle
BYTEARRAY8 5 any pickle containing a bytearray (pickle.dumps(bytearray(...)) uses it by default)
NEXT_BUFFER / READONLY_BUFFER 5 out-of-band buffers (a buffer_callback returning a relocated buffer)

Reproduced pre-fix with benign inputs: pickle.dumps(1.5, protocol=0) and pickle.dumps(bytearray(b"hello"), protocol=5) both crash at Pickled.load(). A security scanner that crashes on certain inputs is itself a problem: in embedding contexts (the fickling.import_hook / scan-API paths) the crash either aborts the host or is swallowed by a caller that then skips the file — analysis is denied exactly where it matters.

Fix

Four opcode subclasses mirroring the existing families:

  • Float(ConstantOpcode) — proto-0 newline-terminated literal, like Int/Long
  • ByteArray8(BinBytes8) — 8-byte length prefix, pushing a bytearray constant, like BinBytes8 does for bytes
  • NextBuffer(Opcode) — pushes an opaque out_of_band_buffer name (the buffer contents live outside the stream, so there is nothing to inspect)
  • ReadOnlyBuffer(Opcode) — marks the top of stack read-only; a documented no-op for the AST

Validation

  • pytest test/test_pickle.py | sample size: N=31 | key metrics: 4 new regression tests fail pre-fix (NotImplementedError: TODO: Add support for Opcode BYTEARRAY8 / FLOAT raised at load) → post-fix 31 passed | result: pass
  • Full suite pytest test/ → 149 passed
  • ruff@0.16.0 check + format --check on both touched files: clean
  • Encode round-trips verified: Pickled.encode() of the new opcodes re-loads to equal values via pickle.loads

Risk / Compatibility

  • Purely additive: four new classes registered by name; no existing opcode or analysis logic touched. Files that previously crashed now analyze normally.

…XT_BUFFER, READONLY_BUFFER)

Fickling's OPCODES_BY_NAME registry covered 64 of the 68 standard pickle
opcodes. The four missing ones raise NotImplementedError from
Opcode.__new__ at load time — before any analysis runs — so
Pickled.load() crashes with a traceback instead of producing a verdict
on perfectly benign pickles: any float literal in a legacy protocol-0
file (FLOAT), any bytearray at protocol 5 (BYTEARRAY8), and out-of-band
buffers (NEXT_BUFFER/READONLY_BUFFER).

Implement the four opcodes mirroring the existing families:
- Float(ConstantOpcode): proto-0 newline-terminated literal, like Int/Long
- ByteArray8(BinBytes8): 8-byte length prefix pushing a bytearray constant
- NextBuffer(Opcode): pushes an opaque out_of_band_buffer name — the
  buffer contents live outside the stream, nothing to inspect
- ReadOnlyBuffer(Opcode): marks the top of stack read-only; a documented
  no-op for the AST
@feiiiiii5
feiiiiii5 requested a review from ESultanik as a code owner September 5, 2026 05:15
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Noting an overlap I should have caught when opening this, so it doesn't cost the maintainers a discovery exercise later.

#320 (opened 2026-08-21, earlier than this PR) also implements BYTEARRAY8:

#320 Bytearray8 this PR ByteArray8
base DynamicLength, ConstantOpcode BinBytes8
name "BYTEARRAY8" "BYTEARRAY8"
priority BinBytes8.priority + 1 BinBytes8.priority + 1
run() overridden inherited from BinBytes

Two consequences if both were merged as they stand, from the current code rather than opinion:

  • Opcode.__init_subclass__ registers OPCODES_BY_NAME[cls.name] = cls (fickling/fickle.py:456), so the second class to import silently wins the BYTEARRAY8 name.
  • Both claim the same ConstantOpcodePriorities value, and ConstantOpcode.new() picks by sorted(..., key=lambda kv: kv[1]) (fickling/fickle.py:582-584) — equal priorities mean which class encodes a bytearray depends on dict insertion order, not on a decision anyone made.

So: I'm happy to narrow this PR to the three opcodes #320 does not touch — FLOAT, NEXT_BUFFER and READONLY_BUFFER — and leave BYTEARRAY8 to #320, which was here first and already covers run(). Say the word (or just say "drop the bytearray commit") and I'll push that within a day; the four are independent classes, so splitting is a rebase, not a rewrite.

For orientation: this PR's regression coverage is the four cases in test/test_pickle.py::TestUnimplementedOpcodeRegression (test_float_proto0, test_bytearray8_proto5, test_next_buffer_and_readonly_buffer_do_not_crash, test_load_never_raises_notimplementederror), which is what "previously raised NotImplementedError on load" refers to.

@feiiiiii5

Copy link
Copy Markdown
Contributor Author

CI is green here, including the optional RE2 mixed-engine job and the 3.11/3.12/3.13 matrix.

The scope is four opcode classes in fickle.py plus round-trip tests in test/test_pickle.py — no changes to how existing opcodes decode. Two of the four (FLOAT, BYTEARRAY8) reproduce on inputs as plain as pickle.dumps(1.5, protocol=0) and pickle.dumps(bytearray(b"hello"), protocol=5), which crash Pickled.load() today.

Is there anything you would like adjusted — naming, the proto-0 float edge cases, or splitting the buffer opcodes into a separate PR — before this can go in?

@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Hi @trailofbits-fickling maintainers — CI is green (2 checks) and the branch is still rebased on master, so this is my second and last nudge.

Recap: OPCODES_BY_NAME covered 64 of the 68 standard pickle opcodes, and the four missing ones raised NotImplementedError at load time, so Pickled.load() crashed on benign inputs like pickle.dumps(1.5, protocol=0) and pickle.dumps(bytearray(b"hello"), protocol=5). The diff is purely additive (four subclasses mirroring the existing families, no analysis logic touched) and pytest test/ is 149 passed with the four regression tests failing pre-fix.

One question rather than a bump: NextBuffer/ReadOnlyBuffer push opaque out-of-band buffer names and are documented no-ops in the AST. If you'd rather they carry the buffer payload for analysis, that's a design call I'd rather hear from you than guess — everything else I believe is ready to merge as is.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant