Add threshold-driven inline vs self-reference storage for FILE values

[alkis]

Builds on apache#3608 (based on brkyvz:fileType, so the delta here is just the two commits on top).

Adds the inline-vs-self-reference decision to the writer, and fixes a few things found while verifying it against the revised spec in apache/parquet-format#603.

Threshold-driven storage

FileValueWriter.write(payload) returns a Placement — either inline bytes, or an offset/size pair — and the caller writes whichever FILE group fields it indicates. The choice comes from ParquetProperties.withFileSelfReferenceThreshold(int) (default: the page size), so the same writing code produces either form:

FileValueWriter.Placement placement = fileValueWriter.write(payload);
if (placement.isInline()) {
  group.add("inline", placement.getInlineBytes());
} else {
  group.add("offset", placement.getOffset());
  group.add("size", placement.getSize());
}

content_type and checksum describe the resolved bytes either way, so they're written identically and consumers see no difference beyond which fields are set.

Payloads are written eagerly, mid-record, before the row group's column chunks flush. That isn't a choice: offset and size are ordinary column values, and once a value reaches a ColumnWriter it's encoded into a buffered page and can't be revised — so a placeholder offset could never be patched up later. Writing at that point also puts payloads in a contiguous run between row groups, which keeps each column chunk contiguous on disk (ConsecutivePartList relies on that when coalescing chunks into one range read).

Fixes

Self-reference AAD keyed on the file offset, not a synthetic counter. Field 6 of the AAD suffix is defined as the offset of the self-reference within the file. The previous selfReferenceOrdinal wasn't recoverable from the file — nothing stores it, so a reader had to walk the preceding values to rebuild it, which defeats the property the offset-based AAD exists to provide (§ "available to a reader without counting preceding values... may therefore resolve a self-reference without decoding the pages it skips"). Since offset is already carried in the value, this also deletes the ordinal parameter from five signatures.

decompressUnknownSize was broken for every codec, not just unframed ones. The drain loop terminated on read() == -1, but NonBlockedDecompressorStream throws IOException("Corrupt file: Zero bytes read during decompression.") once its block is consumed instead of returning -1. Now: grow-and-retry into a dynamically sized buffer (2× compressed size, doubling), per the spec's "decompress into a dynamically sized buffer". All codecs work, including LZ4_RAW, whose maxUncompressedLength can't recover the size. ZSTD exposes no Decompressor at all (createDecompressor() returns null by design) and decompresses only through its stream, which is framed — so that case drains the stream. I tried collapsing both paths into a single stream loop; it fails 8/22 tests, so the split is load-bearing.

2 GiB encrypted-module cap enforced on write, pointing oversized values at external references, rather than silently writing a corrupt length field.

Two pre-existing test compile errors that prevented parquet-hadoop from building at all: TestParquetMetadataConverter used assertEquals/assertTrue with no JUnit import (switched to the AssertJ style used throughout that file), and TestSelfReferenceFileWrite was missing the ParquetReadOptions import. Both fail on be266d31 independently of these changes. Also ran spotless:apply.

Not changed

I'd initially tightened schema validation to require inline whenever offset is declared — uri is optional per value, so a uri+offset+size schema can still emit a self-reference with no inline column chunk to inherit compression/encryption from. But testFileLogicalTypeExternalRangedReferenceWithoutInline and testFileLogicalTypeOffsetWithSize assert that schema is valid, so I reverted to your semantics and documented the write-path expectation instead. Worth a decision either way — happy to follow whichever you prefer.

Testing

parquet-column 679/679 pass; parquet-hadoop 761/762; the 35 self-reference tests all pass. New coverage: threshold routing, offset-keyed AAD round-trips under AES_GCM_V1 and AES_GCM_CTR_V1, tamper detection when an offset is altered, identical payloads at different offsets not being interchangeable, and payloads spanning several buffer doublings (including exact powers of two, empty, and incompressible) across SNAPPY/GZIP/ZSTD/LZ4_RAW.

The one parquet-hadoop failure is testEnumEquivalence → No enum constant Encoding.ALP, an artifact of how I had to build this locally: FileType is unreleased, so I pinned parquet.format.version to a locally installed artifact carrying parquet-format master's parquet.thrift, which defines an ALP encoding the Java Encoding enum doesn't have yet. Unrelated to these changes.

Note this stack still can't go green on CI until a parquet-format release carries FileType (2.13.0's LogicalType union stops at 18), and apache/parquet-format#603 is itself still open.

This pull request and its description were written by Isaac.