Skip to content

[core] Allow primary-key-only deletes for deduplicate tables - #9978

Merged
JingsongLi merged 2 commits into
apache:masterfrom
Akash3121:fix/4702-delete-nullability
Sep 20, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
Akash3121:fix/4702-delete-nullability

Conversation

@Akash3121

Copy link
Copy Markdown
Contributor

Purpose

Allow callers of TableWrite to delete a row from a primary-key table without supplying values for every non-key NOT NULL column.

Previously, TableWriteImpl applied the table's complete nullability validation to every row kind. A direct delete such as the following was therefore rejected when v was declared NOT NULL, even though the deduplicate merge engine only needs the key to identify the row being deleted:

// Schema: pt INT NOT NULL, k INT NOT NULL, v BIGINT NOT NULL
// Primary key: (pt, k)
write.write(GenericRow.ofKind(RowKind.DELETE, 1, 1, null));

This differs from SQL delete paths that first read the existing row and consequently submit a fully populated delete record. Direct  TableWrite  users should not need that additional read when the merge semantics do not consume the omitted fields.

Closes #4702.

Changes

Row-kind-aware nullability validation

 TableWriteImpl  now determines the effective row kind before selecting the nullability checks to apply.

For normal rows, the existing behavior remains unchanged: every field declared  NOT NULL  must still be present.

For eligible  DELETE  rows,  TableWriteImpl  can use a restricted set of required fields instead of validating all non-null columns. The original constructor is preserved and continues to perform full validation, so existing callers do not change behavior unless they explicitly provide the delete-specific field set.

The effective row kind is derived from the default-wrapped row, which preserves support for  rowkind.field . Nullability is still checked against the original input row so configured defaults do not unintentionally bypass input validation.

Conservative scope

The relaxed behavior is enabled only for primary-key tables that satisfy both conditions:

  • The merge engine is  DEDUPLICATE .
  • Cross-partition updates are disabled.

For an eligible delete, the following fields remain required when declared  NOT NULL :

  • All primary-key fields.
  • All configured sequence fields.

Sequence fields remain mandatory because they participate in ordering. A delete with an older sequence value must not remove a newer row.

Full nullability validation is retained for:

  • Inserts.
  •  UPDATE_BEFORE  and  UPDATE_AFTER  rows.
  • Partial-update tables.
  • Aggregate and other non-deduplicate merge engines.
  • Cross-partition update tables, where partition information is required for routing.

Internal serialization support

Skipping the explicit nullability check alone is insufficient. The serializer generated from a logical  NOT NULL  primitive type directly invokes getters such as  getInt  and  getLong ; attempting to serialize a Java  null  through those getters can fail.

For eligible deduplicate tables, the internal key-value value type therefore marks fields that may be omitted from a delete as nullable. Primary keys and sequence fields retain their original nullability.

This is an internal storage/serialization type only. The table's public logical schema and its  NOT NULL  constraints are unchanged, and non-delete rows continue to be validated against the complete logical schema.

The same internal nullable value type is also supplied to lookup merge processing so buffered or spilled delete records are serialized consistently.

Behavior examples

Given this table:

pt INT NOT NULL
k INT NOT NULL
v BIGINT NOT NULL
PRIMARY KEY (pt, k)

A deduplicate table now accepts: 
 write.write(GenericRow.ofKind(RowKind.DELETE, 1, 1, null));

It still rejects a delete with a missing key:
 write.write(GenericRow.ofKind(RowKind.DELETE, 1, null, null));

It also continues to reject null payloads for non-delete row kinds:
 write.write(GenericRow.ofKind(RowKind.UPDATE_AFTER, 1, 1, null));

For a table with a sequence field, the sequence value must still be supplied:

 // Rejected because seq is missing.
 write.write(GenericRow.ofKind(RowKind.DELETE, 1, 1, null, null));
 
 // Accepted: key and sequence are supplied; the payload is omitted.
 write.write(GenericRow.ofKind(RowKind.DELETE, 1, 1, 11, null));

Changelog considerations

This change does not reconstruct the previous non-key values for a direct delete. A changelog producer that exposes the input delete record may therefore contain null values for omitted non-key fields.

That is expected for this API path: the final table semantics are preserved, but callers that require a complete before-image must still provide the previous values or use a changelog mode capable of reconstructing them.

Tests

Added coverage for:

  • A deduplicate primary-key table accepting a delete with only its primary key fields populated.
  • Missing primary-key fields still failing nullability validation.
  • Inserts,  UPDATE_BEFORE , and  UPDATE_AFTER  retaining complete nullability checks.
  • Partial-update tables retaining complete delete validation.
  • Sequence fields remaining required for deletes.
  • An older-sequence delete not removing a newer row.
  • A newer-sequence delete removing the row.
  • Deletes generated through  rowkind.field  receiving the same behavior.
  • Generated update row kinds retaining complete nullability checks.
  • Lookup merge and its spill-capable buffering path supporting partial delete payloads.
  • Cross-partition primary-key tables retaining partition-field validation.

Notes for reviewers

The restriction to non-cross-partition  DEDUPLICATE  tables is intentional:

  • Deduplicate merge semantics select the latest key-value record and do not need non-key values to compute a deletion.
  • Partial-update and aggregate merge functions may consume non-key retract values.
  • Cross-partition updates require partition information for routing and index maintenance.
  • Sequence fields affect record ordering and therefore cannot be omitted safely.

The internal nullable value type is necessary to serialize the omitted fields safely; changing only  TableWriteImpl.checkNullability  would allow the row through validation but could fail later in primitive field serialization.

Akash3121 and others added 2 commits September 18, 2026 22:34
Validate only primary keys and sequence fields for DELETE rows in non-cross-partition deduplicate tables while preserving full nullability checks for other row kinds and merge engines. Use a nullable internal value type so omitted delete payload fields can be serialized safely, including lookup merge paths.

Signed-off-by: Akash Reddy Jammula <akashreddyjammula@gmail.com>
Cover direct and generated deletes, lookup merge, sequence ordering, non-delete row kinds, partial-update tables, and cross-partition validation.

Co-authored-by: GitHub Copilot <copilot@github.com>
Signed-off-by: Akash Reddy Jammula <akashreddyjammula@gmail.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)

Implementation: CLEAN

The change enables direct primary-key-only DELETE records without weakening validation for inserts/updates: only delete rows use the reduced required-field set, routing keys remain present, and sequence fields are retained when needed. The tests cover lookup, cross-partition constraints, sequence ordering, row-kind fields, and rejection of incomplete non-delete rows. I did not find an actionable regression, and CI is green.

@JingsongLi
JingsongLi merged commit f8a28cd into apache:master Sep 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] TableWrite skip null fields check, when RowKind is DELETE

2 participants