Icechunk/Iceberg hybrid for data collections metadata.

[sharkinsspatial]

Hybrid Icechunk/Iceberg Approach

We've had a host of objectives / issues written around analytics scale metadata stores and dealing with heterogeneous arrays. The crux of the problem we are trying to solve is how we can having a single system that manages data and metadata, rather than the separate, disconnected systems and indexes we use today for these separate concerns. Ideally we want a system where data updates and the corresponding metadata are written in single, atomic commit so that the system is logically consistent.

My first experiment for meeting this goal was https://github.com/developmentseed/zarr-datafusion-search which embeds columnar metadata in an Icechunk store as arrays so that everything can be managed using Icechunk's transaction system. This approach works well, scales and meets our objective but it has several downsides

• Required bespoke development and maintenance to create the Datafusion engine which queries this data.
• By conceptually replicating Parquet as 1-d arrays we are not taking advantage of the huge volume of open community and development from the Parquet and Geoparquet communities.
• Arrays do not provide optimal memory layout or encoding for the type of columnar metadata we are representing. This is especially true given how Parquet can leverage dictionary encoding for duplicate values.
• Writing a "row" as chunks across multiple 1-d arrays requires careful synchronization and could lead to Icechunk related scaling limitations for high volume uncoordinated writes.

All in all, it seems like there must be a more elegant solution to this problem that blends the best of both worlds, Icechunk for array data and Iceberg for tabular/columnar data. After a bit of initial investigation and some discussions with @TomNicholas and @paraseba it appears this should be conceptually feasible.

How This Might Work

We have two entities that must stay synchronized, arrays stored in Icechunk and a metadata table describing those arrays stored as Iceberg managed Parquet files. Iceberg normally needs a catalog service to track the current version of the Iceberg table (which is a pointer to an Iceberg metadata.json file). Instead of using Iceberg's catalog service as the source of truth, we will use an attribute inside the Icechunk repo.

Because that attribute is part of the Icechunk repo, a single Icechunk commit updates the arrays and the table pointer together in a synchronized way. A version tag on the Icechunk repo points at both the arrays and the exact table version that describes them.

At a high level we are continuing to use Iceberg's manifest system but we are ignoring its snapshot pointer system and instead using Icechunk's snapshot to point to the correct version of the Iceberg table.

How a hybrid commit works

We use 2 client libraries in conjunction. Each writer is one process using two libraries: an Iceberg client (PyIceberg or similar) that writes table files into the Iceberg store, and an Icechunk client that stages array data and makes the commit with the pointer to the new Iceberg metadata.json.

How do we handle conflicts

Anna's Iceberg client fetches the pointer's new value v11-ben.metadata.json and checks that her operation doesn't clash with it. An append always passes. Then it writes v12-anna.metadata.json whose parent is v11-ben. This file list contains Ben's manifests plus Anna's manifest. Her manifest slots in unmodified because it was written independently — Iceberg fills in the ordering numbers when the file list is read. In this case her Iceberg client only needs to write one small file list and one metadata file. Her old ** v11-anna** is simply abandoned (and can be swept later).

Now, Anna's Icechunk client overwrites the session's Iceberg metadata.json pointer value with the v12-anna path, rebases the session onto Ben's snapshot (which carries her staged chunks forward and re-checks them against what Ben changed) and commits. The pointer now reads v12-anna and the repo snapshot contains Anna's arrays, Ben's arrays, and a table version listing both writers' rows. The final table history is v10 → v11-ben → v12-anna. If a third writer lands during her recovery, she repeats the loop against that writer's version.

One caveat: this automatic merge is for appends. If Anna's operation had depended on something Ben changed (for example she was deleting rows for a granule whose files his operation updated) the validity check fails and her writer must now make a new operation plan to replay her changes on v11-ben rather than v10. Iceberg clients such as iceberg-rust have support for this (though we may need to implement a specialized "catalog" endpoint to make this seamless, more investigation is required).

Modifying Iceberg's behaviour

Since we're using Icechunk rather than Iceberg to manage snaphsots, an Icechunk tag can pin a metadata.json from months ago and every file that version references must stay on disk. Iceberg's built-in cleanup/compaction tools don't know about Icechunk' references — they judge what's deletable by looking only at a table's current version, so they could delete files a tagged version still needs.

Regular commits and even compaction are safe, since they only add files and mark old ones as logically replaced and no files are ever removed. We need to disable Iceberg's cleanup/compaction behaviour.

1. Set the table property gc.enabled=false (which makes expire_snapshots and remove_orphan_files refuse to delete)
2. Explicitly keep write.metadata.delete-after-commit.enabled=false (otherwise ordinary commits quietly prune old metadata files past a count limit)
3. Never register the warehouse with a managed catalog that runs its own maintenance and never call the expiry procedures directly.

Since table properties only constrain well-behaved clients we can also add storage level permissions. Writers get put get on the object with no delete rights.

Using Icechunk as the source of truth

Rather than using Iceberg catalog's behavior for cleanup/compaction we need to manage our own system that relies on Icechunk's tags as the source of truth. This system can periodically run to check every live Icechunk branch and tag, collect every pinned metadata.json, follow each to all the files it references, and remove only what nothing reaches. Most of this differentiation behaviour is implemented in existing Iceberg libraries and just needs to be extended to use our new Icechunk source of snapshots.

A list of related analytics scale metadata store issues for ODD

#309
#351
#247
#291
#173

[TomNicholas]

This is great, thanks @sharkinsspatial

Since we're using Icechunk rather than Iceberg to manage snaphsots, an Icechunk tag can pin a metadata.json from months ago and every file that version references must stay on disk. Iceberg's built-in cleanup/compaction tools don't know about Icechunk' references — they judge what's deletable by looking only at a table's current version, so they could delete files a tagged version still needs.

This seems like a major downside. We should evaluate whether this fragility is worth the advantages of re-using Iceberg. We should also evaluate whether or not combing Icechunk with Lance instead might avoid this downside...