Fix dataset_fields duplication for NULL type causing column_lineage explosion

[kalra-mohit]

DatasetFieldDao.upsert relies on the (dataset_uuid, name, type) unique constraint to turn a repeat upsert of the same field into an UPDATE:

INSERT INTO dataset_fields (uuid, type, created_at, updated_at, dataset_uuid, name, description)
VALUES (:uuid, :type, :now, :now, :datasetUuid, :name, :description)
ON CONFLICT(dataset_uuid, name, type) DO UPDATE SET ...

Problem is standard SQL never treats NULL = NULL as true for uniqueness/ON CONFLICT purposes. Plenty of OpenLineage producers don't report a column type, so any field upserted with type = NULL never matches its own previously-inserted row — a fresh row with a new UUID gets inserted every time. Each of those rows independently accumulates its own column_lineage edges, so what should be one relationship turns into N, and N grows unbounded with every event. That's the "explosion" from #3083 — the reporter saw 1000x+ row bloat.

flowchart TD
    U["upsert dataset_fields
(dataset_uuid, name, type = NULL)"]
    U --> C{"ON CONFLICT (dataset_uuid, name, type)"}
    C --> Old["Before: type stored as raw NULL"]
    Old --> N1["NULL <> NULL, conflict never matches"]
    N1 --> R1["New row inserted every upsert
N rows for 1 field -> N x column_lineage rows"]
    C --> New["After: type stored as COALESCE(:type, 'UNKNOWN')"]
    New --> N2["'UNKNOWN' = 'UNKNOWN', conflict matches"]
    N2 --> R2["Existing row updated
single dataset_fields row retained"]

Loading

Fix: coalesce a NULL type to the sentinel 'UNKNOWN' before insert. Repeated upserts of an unknown-type field now reliably collide with the row already there. No schema/migration change, no risk to rows that already have a real type — this is the fix suggested in the issue itself. One visible side effect: a field's type now shows up as the literal string "UNKNOWN" via the API instead of null when nothing was reported. Updated MarquezAppIntegrationTest#testDatasetWithUnknownFieldType to match.

Verification: ran the actual INSERT ... ON CONFLICT SQL by hand against a real Postgres 16 instance via psql, outside Gradle/Testcontainers (see below for why). Two upserts of the same (dataset_uuid, name) with type = NULL: before the fix, count = 2; after, count = 1, and the returned uuid was identical across both calls, confirming the second call updated rather than inserted.

Correction on that verification, since I got it wrong in an earlier draft of this description and want to correct it plainly rather than leave it: I'd said the Postgres 16 instance I tested against matched "the version pinned by this repo's own tests." That's not accurate. I went and checked api/src/test/java/marquez/PostgresContainer.java (which backs the Testcontainers extension this PR's own new DatasetFieldDaoTest uses), plus docker-compose.yml and docker-compose.db.yml — they all still pin Postgres 14. There's an incomplete, unmerged postgres-16 branch on the remote, and only two unrelated test files (DbRetentionTest, StatsTest) touch Postgres 16 today. So my manual verification used Postgres 16 while this repo's actual pinned test infrastructure runs Postgres 14 — a real version mismatch on my part. It doesn't change the fix's correctness though: the relevant SQL semantics here (NULL never equals NULL for ON CONFLICT/unique-constraint purposes) are identical across Postgres 14 and 16.

Tests:

• DatasetFieldDaoTest (new) — DAO-level: repeated upserts of a null-typed field resolve to the same row/UUID, the stored type becomes "UNKNOWN", and real (non-null) types are unaffected.
• MarquezAppIntegrationTest#testDatasetWithUnknownFieldType_repeatedUpsertsDoNotDuplicateField (new) — emits the same dataset with a null-typed field 5 times, mirroring the issue's repro steps, and asserts only one field comes back, not five.
• testDatasetWithUnknownFieldType — updated to expect the "UNKNOWN" sentinel.

Couldn't run these locally through Gradle — same Testcontainers/Docker API mismatch I keep hitting in this sandbox (bundled docker-java defaults to API v1.32, local engine needs >= v1.40; confirmed it's pre-existing by hitting the identical failure on unmodified RunDaoTest). ./gradlew :api:compileTestJava and :api:testUnit (118 tests) both pass. Given the local DB test gap, the manual psql verification above is what I'm actually leaning on to confirm correctness — would appreciate CI running the full DB-backed suite on this one.

Fixes #3083.

[kalra-mohit]

@merobi-hub whenever you get a chance to take a look, happy to address any feedback!