Skip to content

[core] Fix lookup merge buffer serialization for projected reads - #10549

Merged
JingsongLi merged 1 commit into
apache:masterfrom
reswqa:fix-lookup-merge-buffer-read-type
Oct 16, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
reswqa:fix-lookup-merge-buffer-read-type

Conversation

@reswqa

@reswqa reswqa commented Oct 14, 2026

Copy link
Copy Markdown
Member

Purpose

Fix lookup merge buffer serialization when reading with a projected schema.

LookupMergeFunction.Factory currently creates the candidate buffer using the full valueType, even when the input rows follow a projected readType. Once the number of candidates for a key exceeds lookup.merge-records-threshold, the buffer switches to binary serialization. The mismatch between the serializer schema and the actual row layout can cause read failures when fields are pruned or reordered, including nested projections.

This PR uses readType to derive the buffer schema when it is provided, preserving the projected field order and nested structure. Nullable fields from the internal valueType are retained by matching field IDs. This preserves the delete-record handling introduced in #9978, where omitted values may be null even if the corresponding fields are declared NOT NULL in the read schema.

The original readType is still passed unchanged to the wrapped merge function. Calls without a readType continue to use valueType.

No public API or table file-format changes.

Tests

Added regression coverage for:

  • Pruned and reordered fields during lookup merge buffering.
  • Delete records containing null values for fields declared NOT NULL in the read schema.
  • Nested field projections.
  • Projected batch reads across two data files with lookup changelog enabled and lookup.merge-records-threshold set to 1.

Validated the following test classes with JDK 8:

  • LookupMergeFunctionTest
  • LookupMergeFunctionUnitTest
  • LookupChangelogMergeFunctionWrapperTest
  • LookupMergeTreeCompactRewriterTest
  • PrimaryKeySimpleTableTest

all tests passed, with no failures, errors, or skips.

@JunRuiLee JunRuiLee closed this Oct 14, 2026
@JunRuiLee JunRuiLee reopened this Oct 14, 2026

@JunRuiLee JunRuiLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending green CI.

@JingsongLi
JingsongLi merged commit 7d6d146 into apache:master Oct 16, 2026
34 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants