Specify storage inheritance for FILE self-references - #603
Conversation
alkis
changed the title
| compressed block or the uncompressed bytes. For a compressed block, the complete range | ||
| is supplied to the codec, and its decompressed output is the resolved value. |
There was a problem hiding this comment.
Should we also store the decompressed size somewhere?
There was a problem hiding this comment.
I have thought about this but it complicates the format - we will need framing for the entry. For some compressors this exists already (Snappy, optionally ZSTD), for the rest we can require a dynamically sized decompressor output as discussed here https://lists.apache.org/thread/rfz37kt9bdjw6m7o6mwf8f7nr712c0m8. If there is strong push for uncompressed size we can add a header but that's a larger change.
There was a problem hiding this comment.
Perhaps we cal leave this open to be specified later if needed?
There was a problem hiding this comment.
Agreed. I made that explicit in the spec rather than leaving it unsaid: the decompressed size is not stored, readers rely on the codec's framing where it provides one or decompress into a dynamically sized buffer, and a future revision may add an explicit decompressed size.
| For a self-reference, `offset` points to the beginning of the 4-byte length and `size` | ||
| is the size of the complete encrypted module, including the length, nonce, ciphertext, | ||
| and GCM tag when present. |
There was a problem hiding this comment.
4-byte length limits us to 2GiB payload. Are we expecting use cases with greater size requirements?
There was a problem hiding this comment.
I expect 2GiB payloads to be in their own external file reference not embedded inside parquet.
There was a problem hiding this comment.
And that means we can't use parquet modular encryption, right? Should we note that?
There was a problem hiding this comment.
The opposite — with this PR self-references are modular encryption modules (module type 10, keyed on the inline column key), so modular encryption works for them. This PR removes the prohibition that the merged spec had.
Two limits are worth noting and are now stated:
- The encryption buffer must fit the 4-byte length field, so an encrypted self-reference is capped at 2 GiB as for any other module. Larger values use an external reference.
- The AAD includes the row group and column ordinals, so an encrypted stored representation is bound to one column chunk. A writer must not share one encrypted byte range between column chunks.
| | OffsetIndex | yes | yes (7) | yes | yes | no | no | | ||
| | BloomFilter Header | yes | yes (8) | yes | yes | no | no | | ||
| | BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | | ||
| | Self-Reference | yes | yes (10) | yes | yes | no | yes | |
There was a problem hiding this comment.
Why Page ordinal is not used here? When only a few pages are required to read, we still need to count number of values of skipped pages in order to decrypt a self-referenced value?
There was a problem hiding this comment.
You are right that a derived ordinal is wrong for exactly this reason. I switched the AAD to key on the self-reference offset instead. The offset is a field of the FILE value being read, so the reader has it in hand and never counts values in pages it skips.
Page ordinal would also work for a reader — you know the ordinal of the page you are decoding — but it ties the out-of-band range to page layout, which the rest of this PR deliberately avoids: compression and encryption are properties of the column chunk, not of the page a value happens to land in. Keying on offset keeps the stored representation independent of page boundaries, and the offset already uniquely identifies it within the file.
Key the self-reference AAD on `offset` rather than a derived ordinal, so a reader can resolve a self-reference without decoding the pages it skips. State that the encryption buffer of a self-reference must fit the 4-byte length field, and that larger values must use an external reference. State that the decompressed size is not stored and how readers obtain it. State that an encrypted stored representation is bound to a single column chunk and must not be shared between column chunks.
4 participants
Specify storage inheritance semantics for
FILEself-references:inlinefield for schemas permitting self-referencesinlinecolumn chunkoffsetandsizeover the stored representationDiscussion: https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7