Bidirectional conflict resolution diverges when local tuple has no xmin/commit timestamp (OrioleDB tables)

[mrayva]

Summary

A simultaneous bidirectional conflict (both nodes write the same row within the same replication window) causes the two nodes to diverge instead of converging on one winner, when the target table is an OrioleDB table (or, more generally, any table AM that doesn't populate xmin/xmax).

Root cause

get_tuple_origin() determines a local row's "last modified" timestamp via HeapTupleHeaderGetXmin() on the locally-stored tuple, then looks that XID up in pg_commit_ts. OrioleDB uses UNDO-log/CSN-based MVCC, not xmin/xmax, so when an OrioleDB slot is materialized into a compatibility HeapTuple, xmin is never populated. This is a permanent, by-design characteristic of the AM — see orioledb/orioledb#64, where the OrioleDB maintainers confirmed xmin/pg_xact_commit_timestamp(xmin) will never be meaningful for OrioleDB tables ("Fields other than ctid and tableoid will probably not be added for our tables").

Consequence in spock_handle_conflict_and_apply() (and the analogous DELETE path): when get_tuple_origin() returns found=false, local_ts is left at its zero-initialized value. conflict_resolve_by_timestamp() then always finds the incoming remote tuple "newer" than local (since any real timestamp beats epoch/0). Under a simultaneous conflict, both nodes independently reach that same "remote is newer, apply it" conclusion — so both apply whatever they received from the other side, and they diverge instead of agreeing on a winner.

Reproduction

Two-node Spock cluster with an OrioleDB table on both sides (shared_preload_libraries = 'orioledb, spock'), bidirectional subscriptions both enabled. Fire truly concurrent (not sequentially committed — see note below) conflicting writes:

-- node1
UPDATE oriole_items SET name='node1' WHERE id=1;
-- node2, committed before node1's change has replicated over
UPDATE oriole_items SET name='node2' WHERE id=1;

Result: node1 ends up with 'node1', node2 ends up with 'node2' — permanently diverged, no error raised. (A test loop that commits sequentially with any inter-statement delay will not reproduce this — the second write's local-origin-differs conflict resolution masks the bug. The writes need to actually race, e.g. launched as two background psql processes committed within the same window.)

Suggested fix

When local_origin_found == false, force local_ts to equal the incoming remote timestamp (replorigin_session_origin_timestamp) rather than leaving it at 0. This makes the timestamp comparison read as "equal" instead of "remote always wins," routing the decision to the existing deterministic node-tiebreaker (spock.node.tiebreaker) instead. Both nodes then independently compute the same symmetric comparison and agree on the same winner.

Important: local_origin must be left as InvalidRepOriginId (not also set to replorigin_session_origin) — conflict_resolve_by_timestamp()'s same-origin shortcut treats local_origin_id == remote_origin_id as "not a real conflict, re-applying an update from the same source" and unconditionally applies remote, which reintroduces the same bug through a different path.

local_origin_found = get_tuple_origin(rel, local_tuple,
                                      &(localslot->tts_tid), &xmin,
                                      &local_origin, &local_ts);

if (!local_origin_found)
{
    /*
     * get_tuple_origin() could not determine the local row's real
     * commit timestamp (e.g. OrioleDB tables: no xmin/xmax). Leaving
     * local_ts at 0 makes every remote change look newer than any
     * local one, so a simultaneous bidirectional conflict resolves
     * to apply_remote on both nodes independently -> divergence.
     * Force the comparison to "equal" so it routes to the
     * deterministic node-tiebreaker instead. local_origin is
     * deliberately left invalid so the same-origin shortcut below
     * doesn't short-circuit into the same bug.
     */
    local_ts = replorigin_session_origin_timestamp;
}

apply = try_resolve_conflict(rel->rel, local_tuple,
                             remotetuple, &applytuple,
                             local_origin, local_ts,
                             &resolution);

This same fix is needed in three places that can hit a row with no determinable local timestamp: spock_handle_conflict_and_apply() (covers both plain UPDATE/UPDATE and INSERT-that-becomes-UPDATE-on-conflict), and spock_apply_heap_delete()'s origin-check branch (DELETE vs. concurrent update). Verified fix + patch against current main available on request; happy to open a PR if useful.

Environment

• Spock built from main (reports as version 6.0.0)
• PostgreSQL 18.6
• OrioleDB built against the same PG18.6 base, orioledb.so + spock.so both in shared_preload_libraries
• Two-node bidirectional cluster, both subscriptions active

Related

• ERROR: virtual tuple table slot does not have system attributes orioledb/orioledb#64 — confirms xmin/commit-timestamp is fundamentally unavailable on OrioleDB tuples by design, which is the underlying reason get_tuple_origin() can't do its job here.
• conflict reported on subscriber node after re-subscription due to missing OriginID/OriginTimestamp #347, Spurious entries in spock.resolutions for update from new origin (5.0.4) #325, Spock Multi-Master: spock.resolutions Filling with update_update Despite Single-Writer Setup #349 — same general symptom class ("row has no origin/timestamp → conflict resolution goes wrong"), but from a different trigger (tablesync/synchronize_data losing origin metadata on resync, not steady-state AM incompatibility).