Repository navigation
Conversation
sundapeng
changed the title
e24bc8a to
78eac7b
Compare
84d1500 to
30a7dac
Compare
When the options carry io-cache endpoints, for example vended with a REST
table token, the OpenDAL OSS FileIO now reads immutable files through a
cache target chosen by file type and keeps every other request on the
OSS endpoint. The rules are the same as in the Java Jindo FileIO and
pypaimon.
- io-cache.endpoint names one target. io-cache.targets with
io-cache.target.<name>.endpoint (and optional .path-style-access) names
several, and io-cache.routes maps file types to them. io-cache.whitelist
and io-cache.policy keep their meaning, and io-cache.origin.endpoint
names the OSS endpoint when fs.oss.endpoint points at a cache for older
clients. io-cache.enabled set in the catalog options overrides the token.
- Only read, stat and exists requests use a target. Writes, including
OutputFile::exists, listings, deletes, renames, copies and non-oss paths
stay on OSS, and a client-side dlf.oss-endpoint turns routing off.
- Files rewritten in place (FileType::is_mutable) and metadata named by a
sequence id never use a target.
- Data files are recognized by name: a data or changelog file prefix,
including a custom one, followed by {uuid}-{count}.{extension}.
Manifests, indexes and statistics share the {uuid}-{count} shape but
have no extension, so a custom prefix such as manifest- cannot mix them
up; the manifest sidecar is excluded explicitly. Other names keep their
FileType, and names Paimon does not write stay on OSS.
- Errors from a target are returned as they are, without retrying on OSS.
- Adds fs.oss.path-style-access for the OSS endpoint and documents the
options in getting-started.
The routing cases in testdata/io_cache are shared with the Java Jindo
FileIO and pypaimon.
30a7dac to
ef2a480
Compare
Some cache endpoints, such as an OSS accelerator, also take writes and send them to OSS over their own write path, whose bandwidth adds to the direct one. With write in io-cache.policy, OutputFile writers of cacheable files (data files, manifests, indexes) use the file type's target. OutputFile::exists, the existence check a writer makes, stays on origin, as do deletes, renames, copies, listings and writes of snapshots, hints and temp files. A policy with only write routes writes only.
sundapeng
changed the title
Replace the JSON test vectors with assertions grouped by topic in the cache_routing tests, and remove testdata/io_cache.
sundapeng
changed the title
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 16bc555. Per-file-type OSS routing has a practical end-to-end use with REST-vended endpoints, but commit recovery currently trusts stale cache metadata and can publish a snapshot referencing cleaned data.
Validation: all 167 IO tests passed on the PR head, including an added real Parquet table write/commit/scan probe with both meta,read and meta,read,write; hiding the newly committed snapshot in the cache did not prevent discovery. The automatic merge with current main b262b5f is clean and all 3,772 core tests passed (6 ignored), including that probe. An additional actual commit/truncate/expire/delete/replay probe reproduced the missing-file recovery regression below. Routing rules were also compared with Java #10350 at 8bb021cd. No live OSS/cache cluster was available.
| format!("Failed to check existence of '{path}'"), | ||
| ) | ||
| }) | ||
| op.operator(OpClass::Meta) |
There was a problem hiding this comment.
[P1] Keep recovery validation on origin
TableCommit::check_recovery_entries uses this method to check prepared files before replaying a commit whose identity has expired. With io-cache.policy=meta,read, that writer safety check now uses the target. Deletes go only to origin, so a cache can still return a positive HEAD after cleanup. I reproduced: writer A commits identifier 7; B truncates; snapshot 1 is expired and A's data file is deleted from origin; the cache retains the old object/status; replaying A returns Ok(()) and publishes snapshot 3 referencing the absent origin file. There is no subsequent origin existence check. This can turn a recovery into a broken committed table. Please use the origin-only FileIO for recovery existence validation (both recovery APIs), and add a stale-positive-cache regression. In the same probe, changing this safety check to origin_only().exists rejects the replay and leaves snapshot 2 current.
There was a problem hiding this comment.
Thanks, this is right, and the Java client has the same check (TableCommitImpl.checkFilesExistence before replaying committables), as do manifest cleanup and changelog expiration there. Rather than special-casing each caller, ed39f90 gives exists its own token in io-cache.policy: meta now covers file status only, and FileIO::exists / InputFile::exists use a target only when the policy lists exists. Without it, including the meta,read policy of your probe, both recovery paths check existence on the origin. The server is expected to list exists only for targets that answer existence as OSS does, that is, ones that keep no metadata of deleted objects.
test_exists_ignores_a_cache_that_kept_a_deleted_file covers a target that still holds a file the origin no longer has, at the FileIO level. apache/paimon#10350 (421917d084) and apache/paimon#10351 (26fc70998b) apply the same rule.
exists answers whether a file is still there: commit recovery checks the files it replays. A cache that keeps the metadata of deleted objects answers yes after the file was deleted on origin, so a replay could publish a snapshot that references a deleted file. meta now covers file status only; FileIO::exists and InputFile::exists use a target only when io-cache.policy lists exists, as in the Java and Python clients.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS. The new exists policy fixes the previous stale-positive recovery finding: a real Parquet write/commit/read, truncate, snapshot expiration, origin deletion and replay probe now rejects both recovery APIs and leaves snapshot 2 current even while the target retains the deleted data. All 188 IO, REST-token and snapshot tests pass.
A separate actual Format Table read reproduces the stale-row regression below: after replacing manifest.parquet on origin, a direct scan reads [2] while the routed scan reads [1]. No live OSS/cache deployment was available. The same Format Table regression reproduces after clean integration with main dd25596; a temporary origin-only FormatTable read control makes both probes pass.
| } | ||
| match FileType::classify(path) { | ||
| FileType::Data => None, | ||
| file_type => Some(file_type), |
There was a problem hiding this comment.
[P2] Keep mutable external Format Table files on origin
This fallback trusts the broad FileType::classify result: any name containing "manifest" or starting "stat-" becomes Meta, even when it is an ordinary external data file rather than a uniquely named Paimon object. Format Table discovery accepts such names and uses this FileIO. I reproduced with a real Parquet Format Table at oss://bkt/db.db/review_external/manifest.parquet and independent OSS mock origin/cache endpoints: retain the file containing id=1 in the target, replace the origin file with id=2, then scan again with io-cache.policy=meta,read and the default whitelist. The direct origin scan returns [2]; the routed scan returns [1], and request logs show its Parquet GETs going to the target. No local block cache is enabled. This silently returns stale data after a supported external replacement. Please require actual immutable Paimon metadata/index filename shapes before routing, or bypass this routing for mutable Format Table files, and add the overwrite/read regression.
There was a problem hiding this comment.
Thanks, agreed: the fallback trusted names Paimon does not write. 33f4fb9 sends metadata and index files to a target only when the whole name has the shape Paimon writes: (manifest|manifest-list|index-manifest|stat)-{uuid}-{n}, manifest-{uuid}-{n}.avro.sidecar, index-{uuid}-{n} (bucket, deletion vector and primary-key index files) and <kind>-global-index-{uuid}.index; file indexes only through the data-file shape. Any other name, including manifest.parquet and stat-2024.parquet in a Format Table, goes to the origin.
test_format_table_file_named_like_a_manifest_uses_origin mirrors your probe (target holds id=1, origin id=2): the routed read returns id=2 and the target sees no request; the type tests add those names as well. apache/paimon#10350 (1e6598aafb) and apache/paimon#10351 (da72147920) apply the same rule.
…s them
The broad FileType classifier also matched names such as manifest.parquet,
stat-2024.parquet or foo.index, which a Format Table may hold as ordinary
files and replace in place, so a routed scan could return the old file
from a cache. A metadata or index file now uses a target only under the
exact name Paimon writes: manifest-, manifest-list-, index-manifest- and
stat-{uuid}-{n} (a manifest also with .avro.sidecar), index-{uuid}-{n},
and {kind}-global-index-{uuid}.index. Data files and their file indexes
keep the existing name rule; every other name goes to the origin.
test_format_table_file_named_like_a_manifest_uses_origin replaces such a
file on the origin while a target still holds the old one.
# Conflicts: # crates/paimon/src/io/file_io.rs
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 98ae24a. The previous exists-policy and manifest.parquet findings are fixed. Independent actual Parquet write/commit/read and both recovery API probes pass with meta,read and meta,read,write; an origin-deleted file retained by a separate cache is rejected and snapshot 2 remains current.
Validation: all 3,913 core tests pass (6 ignored), including the original routing/REST/snapshot tests. Additional real Format Table reads against independent origin/cache endpoints confirm manifest.parquet now returns current rows, but a valid external file named data-{uuid}-1.parquet still returns stale rows; the origin-only control returns current rows. An actual PK input-changelog commit and incremental read also show the generated changelog manifest PUT/GET bypassing the configured metadata target. Both remaining findings reproduce after clean integration with main 2ec5871. No live OSS/cache deployment was available.
Please also update the getting-started io-cache.policy documentation: meta now means file status only; exists is a separate supported token.
| if temp || FileType::is_mutable(path) { | ||
| return None; | ||
| } | ||
| if is_data_file_name(name, data_prefixes) { |
There was a problem hiding this comment.
[P2] Keep mutable Format Table files on origin even with Paimon-shaped names
A whole-name match still does not establish that an external Format Table object is immutable. Format Table discovery accepts copied/imported Paimon Parquet filenames and upstream jobs may replace those external files under the same name. I reproduced with a real Parquet Format Table at oss://bkt/db.db/review_external/data-123e4567-e89b-12d3-a456-426614174000-1.parquet: the independent cache retains id=1 and origin contains its replacement id=2. With meta,read and default whitelist, the routed table scan returns [1]; the same table through origin_only returns [2]. No local block cache is enabled. The manifest.parquet control now correctly returns [2]. Please preserve the ownership distinction at the Format Table read boundary (for example origin-only IO for mutable external sources); filename shape alone leaves the silent stale-row regression.
There was a problem hiding this comment.
Agreed: a Paimon-shaped name does not make a Format Table file immutable. 129fad9 gives a Format Table the origin-only view of its FileIO in Table::new and Table::from_resolved_schema, so its scans, file status lookups and writes all use the origin whatever io-cache.policy says; other tables keep per-request routing.
format_table_reads_replaced_files_from_origin_with_io_cache mirrors your probe: the target keeps id=1 at review_external/data-{uuid}-1.parquet and the origin holds the replacement id=2. The scan returns [2], an append also stays on the origin, and the target sees no request. Without the change the same test returns [1].
There was a problem hiding this comment.
Following up: after another look I reverted 129fad9 in b85e26b and leave this decision to the catalog that vends the targets.
A Format Table is a Hive-style table whose files scheduled jobs may replace in place, but whether that returns old rows depends on the target. One that forwards every request to OSS, such as an OSS accelerator, or a cache that revalidates with OSS, never returns the replaced content, and routing Format Table reads through it is useful. Only a target that serves cached content without asking OSS can return the old copy, and FileIO cannot tell the two kinds apart, while the REST catalog knows which kind it vends. It already resolves io-cache.policy and io-cache.whitelist per table, database and catalog, so it can leave the io-cache options out for such tables, the same way exists is only vended for targets that answer it as OSS does.
The getting-started section now says that the catalog decides per table whether a Format Table uses a target. The changelog manifest fix (b63dae4) and the docs fix (6ed10b4) stay.
There was a problem hiding this comment.
I rechecked this on b85e26b through the actual FileSystemCatalog create/load/scan APIs with catalog-level io-cache.enabled=true and the documented meta,read policy. After replacing an external Parquet object on origin from row 7 to row 9 while the target retains row 7, the table scan still returns row 7; listing reaches origin, but the data GETs reach the target. The same table with origin-only FileIO returns row 9. No local block cache is enabled.
The proposed per-table catalog decision is absent on the built-in paths: FileSystemCatalog shares its catalog FileIO across tables (filesystem.rs:241–245), and external REST tables explicitly skip table tokens (data_token_enabled && !is_external, rest_env.rs:373) and build FileIO from catalog options (513–518). Thus a mixed catalog using the advertised read cache cannot safely load these external files through a per-table token override. Please retain a safe default for mutable external sources or wire an actual per-table decision before enabling their cache reads, with a catalog load/replace/read regression.
There was a problem hiding this comment.
Agreed, the per-table decision has no way through these loads. 194d85d gives a Format Table the origin-only view of its FileIO again, in Table::new, Table::from_resolved_schema and copy_with_resolved_schema, so FileSystemCatalog tables, REST external tables and token-backed tables all read, stat and write their files on the origin whatever io-cache.policy says. ObjectTable keeps an origin-only view as well, since other writers manage its objects too. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin with io-cache.origin.endpoint.
format_table_reads_replaced_files_from_origin follows your catalog probe: FileSystemCatalog create/load/scan with catalog-level io-cache.enabled=true and meta,read, the target keeping row 7 and the origin holding its replacement, row 9, under a Paimon-shaped name. The loaded table, a copy_with_options copy and a from_resolved_schema table all read 9 and the target sees no request; without the change they read 7. object_table_reads_objects_from_origin covers an Object Table's FileIO.
| // in place. | ||
| static META_FILE: LazyLock<Regex> = LazyLock::new(|| { | ||
| Regex::new(&format!( | ||
| r"^(?:(?:manifest|manifest-list|index-manifest|stat)-{UUID}-[0-9]+|manifest-{UUID}-[0-9]+\.avro\.sidecar)$" |
There was a problem hiding this comment.
[P2] Recognize the changelog manifest names produced by TableCommit
TableCommit names input-changelog manifests manifest-{uuid}-changelog-{count} (and can add .avro.sidecar), but this regex only accepts manifest-{uuid}-{count}. Those immutable metadata files therefore always go to origin. In an actual PK table with changelog-producer=input and io-cache.policy=meta,read,write, the commit and incremental Changelog read send both PUT and GET for manifest-...-changelog-0 to origin, while the ordinary manifest, manifest lists and changelog data use the target. This leaves a normal CDC metadata path outside the configured route. Please include the actual writer naming variant and add a real changelog commit/read routing regression, including its sidecar form.
There was a problem hiding this comment.
Thanks, b63dae4 accepts manifest-{uuid}-changelog-{count} and its .avro.sidecar form (the infix only for manifests, not manifest lists). test_input_changelog_manifests_use_the_io_cache_target commits a primary-key table with changelog-producer=input and manifest.sidecar.enabled=true under meta,read,write, reads the changelog incrementally, and checks that the changelog manifest PUT and GET and the sidecar PUT go to the target and none reach the origin; it fails on the previous rule. apache/paimon#10350 (bd41c99320) and apache/paimon#10351 (b449c5e266) accept the same names, so Java and Python readers of these tables route them alike.
The getting-started section now lists exists as its own policy token, with meta covering file status only (6ed10b4).
A Format Table may hold files that other writers copy in or replace under
the same name, so even a Paimon-shaped name such as
data-{uuid}-1.parquet does not make its content immutable, and a routed
scan could return the old copy a cache still holds. Table::new and
Table::from_resolved_schema now give a Format Table the origin-only view
of its FileIO, so its reads, file status lookups and writes all use the
origin whatever io-cache.policy says; other tables keep the per-request
routing.
format_table_reads_replaced_files_from_origin_with_io_cache reproduces
the review probe: the target keeps id=1, the origin holds the
replacement id=2, the scan returns [2], an append stays on the origin,
and the target sees no request.
TableCommit names input-changelog manifests
manifest-{uuid}-changelog-{count}, optionally with .avro.sidecar, but the
metadata name rule only accepted manifest-{uuid}-{count}, so these
immutable files always went to the origin. The rule now accepts the
-changelog infix for manifests (not for manifest lists).
test_input_changelog_manifests_use_the_io_cache_target commits a
primary-key table with changelog-producer=input and manifest sidecars
under meta,read,write, reads the changelog incrementally, and checks
that the changelog manifest and its sidecar are written and read
through the target and never reach the origin.
meta now covers file status only and exists is a separate token, which the OSS Cache Routing section still described as one.
This reverts 129fad9. Format Table files may be replaced in place by scheduled jobs, but whether that can return old content depends on the target: one that forwards every request to OSS, such as an OSS accelerator, never does. The REST catalog that vends the targets knows which kind it vends and already resolves the io-cache options per table, database and catalog, so it decides whether a Format Table uses a target, as it does for exists. FileIO routes Format Table files by name like those of any other table again; the docs say who decides.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Per-request routing has end-to-end value for separating immutable data/cache traffic from mutable metadata and combining OSS endpoint capacity.
Implementation: FINDINGS. Two new routing gaps below were reproduced through real FileSystemCatalog create/load/write/commit/read APIs. The existing mutable Format Table stale-read finding also still reproduces on this head; I posted the current catalog-path evidence in its original thread.
Validation on this head integrated with current main: 173 IO tests, 8 REST token FileIO tests, snapshot commit/discovery and actual input-changelog manifest/sidecar tests all pass (183 standard targeted tests). An additional real commit-recovery probe passes: after the original snapshot and data are deleted on origin while retained by the target, both recovery APIs reject the replay and the latest snapshot remains unchanged. This confirms the exists-policy fix. Independent writer-routing and external-Parquet replacement assertions fail as described. These use local HTTP OSS-compatible endpoints and actual Arrow/Parquet/BLOB data; a live OSS accelerator/STS deployment was not available.
| fn data_prefixes(props: &HashMap<String, String>) -> Vec<String> { | ||
| let mut prefixes: Vec<String> = DEFAULT_DATA_PREFIXES.map(String::from).to_vec(); | ||
| for key in [DATA_FILE_PREFIX, CHANGELOG_FILE_PREFIX] { | ||
| if let Some(prefix) = props.get(key).filter(|prefix| !prefix.trim().is_empty()) { |
There was a problem hiding this comment.
[P2] Pass the loaded table's custom prefixes into the router
These props come from FileIO/catalog or REST-token options, but normal table data-file.prefix / changelog-file.prefix settings live in the loaded schema. FileSystemCatalog passes its shared FileIO unchanged to Table::new, and TableWrite obtains the prefixes from that schema, so the table-specific values never reach this parser. I created and loaded a real catalog table with schema data-file.prefix=custom-, enabled meta,read,write on a data target, and wrote/committed/scanned a row: all PUT/GETs for custom-<uuid>-0.parquet went to origin, while the identical default-prefix control used the target. This silently defeats the advertised custom-prefix/data routing unless each table prefix is manually duplicated into catalog/token properties. Please propagate the resolved table prefixes into its FileIO routing context and cover a schema-only custom prefix through the actual catalog API.
There was a problem hiding this comment.
Thanks, 6341ba4 gives each table a view of its FileIO that carries the data-file.prefix and changelog-file.prefix of its schema, built in Table::new and Table::from_resolved_schema, and every request passes them to the router, through RESTTokenFileIO as well. copy_with_options and copy_with_resolved_schema add the prefixes of the new schema, so files written under a dynamic prefix use the target while earlier files keep using it.
data_files_named_with_the_schema_prefix_use_the_target creates and loads a FileSystemCatalog table with data-file.prefix=custom- only in its schema, under meta,read,write and the data whitelist, then writes, commits and scans it: the PUT and GET of custom-<uuid>-0.parquet go to the target and none to the origin, and the same holds for a copy with a dynamic- prefix. It fails without the change. test_token_file_io_routes_files_named_with_table_prefixes covers the REST token path.
| // Data files are named {prefix}{uuid}-{count}.{extension}; this matches what follows the prefix. | ||
| const UUID: &str = "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"; | ||
| static DATA_FILE_SUFFIX: LazyLock<Regex> = LazyLock::new(|| { | ||
| Regex::new(&format!(r"^{UUID}-[0-9]+\..+$")).expect("valid data file pattern") |
There was a problem hiding this comment.
[P2] Recognize the immutable names emitted by the supported writers
The {uuid}-{count}.{extension} restriction excludes actual Paimon data files, even with the default data- prefix: the postpone writer emits data--u-<commit-user>-s-<write-id>-w-<uuid>-0.parquet (TableWrite::create_postpone_writer), and ManagedBlobWriter emits data-<uuid>.managed.blob. Through real catalog tables with meta,read,write and a data target, a bucket=-2 primary-key write sent its Parquet PUT to origin, and a primary-key BLOB table sent its managed-pack PUT and subsequent GET to origin; the ordinary Parquet control used the target. These unique, immutable payloads therefore miss both the cache reads and the extra write path/quotas this feature is intended to provide. Please recognize the existing writer formats (and their relevant companions) with routing tests driven by real writers, while preserving the separate protection needed for mutable external Format Table files.
There was a problem hiding this comment.
Thanks, 6c2f5c4 accepts the postpone writer's {prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension} and managed BLOB packs {prefix}{uuid}.managed.blob as data files, with the .index and .blobref files named after them. I also went through the other names the crate writes on current main (data, changelog and blob files, bucket and deletion vector indexes, global indexes including vector, full-text and Lumina, manifests, statistics); they already match.
postpone_bucket_files_use_the_target writes a bucket=-2 primary-key table and managed_blob_packs_use_the_target writes and reads a primary-key BLOB table through FileSystemCatalog under meta,read,write: the postpone file and the packs are written and read on the target and never on the origin. Both fail without the change. Format Table files keep using the origin whatever their names (194d85d).
…che routing Other jobs may replace the files of a Format Table or an Object Table in place under the same name, so a Paimon-shaped name does not make them immutable. Leaving the choice to the catalog that vends the targets does not hold: FileSystemCatalog shares one FileIO across its tables, and REST external tables build theirs from the catalog options without a table token, so a catalog with a read cache had no way to exempt them. Table::new, Table::from_resolved_schema and copy_with_resolved_schema give a Format Table the origin-only view of its FileIO again, and ObjectTable keeps one for its objects, so their reads, file status lookups and writes use the origin whatever io-cache.policy says. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin with io-cache.origin.endpoint. format_table_reads_replaced_files_from_origin creates and loads a Format Table through FileSystemCatalog with catalog-level io-cache and meta,read. The target keeps row 7 and origin holds its replacement, row 9, under a Paimon-shaped name; the loaded table, a copy and a table built from the resolved schema all read 9, and the target sees no request. Without the change they read 7. object_table_reads_objects_from_origin does the same for an Object Table's FileIO.
A table's data-file.prefix and changelog-file.prefix live in its schema, but the io-cache router only read them from the FileIO options, so a table with a custom prefix sent every request for its data files to origin unless the prefix was copied into the catalog or token options. Each table's view of its FileIO now carries the prefixes of its schema, and every request passes them to the router, through RESTTokenFileIO too. copy_with_options and copy_with_resolved_schema add the prefixes of the new schema, so files written under a dynamic prefix use the target while earlier files keep using it. data_files_named_with_the_schema_prefix_use_the_target writes, commits and scans a FileSystemCatalog table whose schema alone sets data-file.prefix to custom-, under meta,read,write and the data whitelist: the PUT and GET of its data files, and of a copy with a dynamic- prefix, go to the target and none to origin. test_token_file_io_routes_files_named_with_table_prefixes covers the REST token path.
…ting
The postpone writer names its files
{prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension} and the
managed BLOB writer names its packs {prefix}{uuid}.managed.blob. Both are
written once under a unique name, but neither matched
{prefix}{uuid}-{count}.{extension}, so their reads and writes always used
origin. Both are now data files, with the .index and .blobref files named
after them. The other names the crate writes on current main (data,
changelog, blob, bucket and deletion vector indexes, global indexes,
manifests, statistics) already match.
postpone_bucket_files_use_the_target writes a bucket=-2 primary-key table
and managed_blob_packs_use_the_target writes and reads a primary-key BLOB
table through FileSystemCatalog under meta,read,write: the postpone file
and the packs are written and read on the target and never on origin.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 6c2f5c4. Per-request OSS endpoint routing has end-to-end value. The previous mutable Format Table stale-read finding and schema-prefix / postpone / managed-pack naming findings are fixed on their covered paths. Two additional production cases remain below.
Validation on the clean integration with main 272cce9: 174 IO tests, 9 REST-token FileIO tests, 5 real catalog/table routing tests, snapshot commit/discovery, and actual input-changelog routing pass (190 tests). Three independent real HTTP/Parquet/BLOB writer, external replacement, and commit-recovery probes also pass (193 total). Two further actual catalog/write/read probes reproduce the findings: old-prefix routing after persisted ALTER/reload, and stale external source bytes permanently copied into a managed BLOB pack. Retaining the old prefix in the reloaded FileIO context fixes the first probe; using origin-only FileIO for descriptor-source ingestion fixes the second. The temporary controls were restored after verification.
These checks use local HTTP OSS-compatible endpoints and actual persisted Arrow/Parquet/BLOB data. A live OSS accelerator / production STS deployment was not available.
| if options.is_format_table() { | ||
| return file_io.origin_only(); | ||
| } | ||
| file_io.with_file_prefixes([options.data_file_prefix(), options.changelog_file_prefix()]) |
There was a problem hiding this comment.
[P2] Preserve retained data-file routing across table reloads
Only the latest loaded schema's prefixes reach this view. I reproduced with a real FileSystemCatalog and HTTP OSS endpoints under meta,read,write / whitelist=data: create data-file.prefix=older-, write/commit/read through the target, ALTER that option to newer-, then load, append, and load again. The fresh scan returns both rows correctly, but the old committed older--0.parquet GET goes only to origin; target sees no request for it. These files remain immutable Paimon data, so a supported prefix change/restart silently loses the configured cache/accelerator traffic allocation for existing data. The same risk exists for dynamic prefixes whose write options are not persisted. The current test retains one FileIO clone that remembers old prefixes, which does not exercise fresh loads. Please recover trusted routing context for committed files across reloads, rather than only the latest in-memory schema prefixes, and add the persisted ALTER/reload regression. Retaining older- in the reloaded context makes the actual endpoint assertion pass. No row-data loss was observed in this case.
| }), | ||
| FileIOBackend::Storage(storage) => { | ||
| let (op, relative_path) = self.create_static(path)?; | ||
| let (op, relative_path) = self.create_routed_static(path, &self.file_prefixes)?; |
There was a problem hiding this comment.
[P2] Read independently owned BLOB ingestion sources from origin
The new routing also reaches BlobFormatWriter's external descriptor source reads through UriInput, using only the source basename to infer immutability. I reproduced with policy=read / whitelist=data: external oss://bkt/import/data--1.png contains NEW on origin and retained OLD on target; pass BlobDescriptor(uri,0,3) as a PK BLOB input and perform normal TableWrite / prepare / commit. The source GET reaches target only, and the newly committed managed pack permanently contains OLD. Reading that pack through an origin-only table still returns OLD, so this is incorrect persisted payload. External descriptor URIs are supported inputs (see primary_key_blob_copies_input_descriptor_into_managed_pack) and have no immutable-source contract; the Format/Object Table guards do not protect this source owned outside the normal table. Please use origin for independently owned descriptor-source ingestion in both BLOB writer paths, or establish trusted source ownership before routing. Preserve routing for the new immutable outputs. An origin-only input FileIO control copies NEW and passes the actual write/read regression.
2 participants
Purpose
The paimon-rust side of apache/paimon#10350: when the options carry OSS cache endpoints, for example vended with a REST table token, the OpenDAL OSS FileIO sends each request to a cache endpoint or to the origin, by operation and file type. The options and rules are the same as in apache/paimon#10350 (Java) and apache/paimon#10351 (pypaimon); the table rules added in this round (Format and Object Tables on the origin, the table's own file prefixes, postpone and managed BLOB names) are being ported to both.
How a request is routed
Every request goes either to a cache target or to the origin,
io-cache.origin.endpoint(by defaultfs.oss.endpoint):oss://, operations not listed inio-cache.policy(read,metafor file status,exists,write), temporary files, files rewritten in place and files named by a sequence id (snapshot-N,schema-N,changelog/changelog-N) go to the origin.data-file.prefixandchangelog-file.prefix, followed by{uuid}-{count}.{extension}, the same after the postpone writer's-u-{commit user}-s-{write id}-w-, or{uuid}.managed.blobfor a managed BLOB pack; its file index adds.indexand its BLOB references.blobref. Metadata ismanifest-,manifest-list-,index-manifest-orstat-followed by{uuid}-{count}, a changelog manifest fromTableCommitismanifest-{uuid}-changelog-{count}, and a manifest may add.avro.sidecar; a bucket index isindex-{uuid}-{count}; a global index is<kind>-global-index-{uuid}.index. Any other name, such as an external Format Table file calledmanifest.parquet, goes to the origin.io-cache.whitelist. Withio-cache.routes, the first rule naming the type picks the target; without it, the first target with an endpoint takes every type. A type no rule names goes to the origin.A target only ever sees files written once under a unique name, so it can be a read cache that keeps metadata.
existsis the exception: Paimon uses it to ask whether a file is still there, for example before replaying a commit on recovery, and a cache that keeps the metadata of deleted objects answers yes after the file is gone. Soexistsuses a target only when the policy lists it, which suits a target that answers existence as OSS does. The origin sees everything else, so it must answer every request as OSS does: OSS itself, or an endpoint that keeps no metadata cache, such as an OSS accelerator. Format Tables and Object Tables never use a target: other jobs may replace their files in place under the same name, whatever the name looks like. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin.meta,read,exists,writemeta,readUsage
A read cache in front of the bucket
Reads and file status of immutable files use the cache; existence checks and everything else use OSS.
An endpoint that also takes writes
Data files are read, written and checked through the OSS accelerator, whose read, write and request paths to OSS add to the direct quotas; everything else uses OSS.
existsis listed because the accelerator keeps no metadata cache.The accelerator as origin, a read cache for data
Data files and global indexes are read through the cache. Snapshot probes, hint files, manifests, listings and all writes go through the accelerator.
The accelerator as origin, data straight to OSS
Data files are read and written on the OSS endpoint; metadata and the other request-heavy traffic go through the accelerator, so the two quotas add up.
Two targets by file type
Manifests and statistics are read through the accelerator, data files and global indexes through the cache, and the rest, existence checks included, uses OSS.
Options
The same options and values as in apache/paimon#10350, vended with the table token or set in the catalog options. None existed in paimon-rust before this PR.
io-cache.enabledio-cache.policyread,meta(file status),exists,write;noneturns it off. Whole tokens, case-insensitive.io-cache.whitelistmeta,data,bucket-index,global-index,file-index, or*(the default).io-cache.endpointdefault.io-cache.targetswithio-cache.target.<name>.endpointio-cache.target.<name>.region,io-cache.target.<name>.path-style-accessdefaultincluded. OpenDAL signs without a region, so.regionis accepted and ignored.io-cache.routesio-cache.origin.endpointfs.oss.endpoint, which may point at a cache for clients without routing.dlf.oss-endpoint(turns routing off) and the table optionsdata-file.prefix/changelog-file.prefix(read from the table schema, and from the catalog or token options, to recognise data files) are existing options this PR only reads.Brief change log
io/cache_routing.rsparses the options and picks the endpoint of each request.io-cache.target.<name>.regionis accepted but not needed, since OpenDAL signs OSS requests without a region.FileIOkeeps one OpenDAL operator for the origin and one per target. Reads,statandexistsfollow the read, meta and exists rules, andOutputFilewrites follow the write rule;OutputFile::exists, the check a writer makes, uses the origin.RESTTokenFileIO:io-cache.enabledset in the catalog options overrides the token.fs.oss.path-style-accessfor the OSS endpoint.Tests
cache_routing.rs: the endpoint of each request with one and two targets, the policy (existson its own included), whitelist, routes and endpoint options, the type of each file name (custom prefixes that collide with reserved names, a table's own prefixes, postpone and managed BLOB names with their.indexand.blobreffiles, entropy and postpone directories, sidecars, temporary names, metadata directories, external names that only look like metadata or indexes such asmanifest.parquet), and option parsing.file_io/cache_routing_tests.rsruns a FileIO against local OSS test servers: which operations reach the target with and withoutwriteandexistsin the policy, a target that still holds a file the origin deleted, a Format Table file namedmanifest.parquetread from the origin while the target holds an older copy, per-target operators, errors from a target, the local block cache.snapshot_manager.rs: committing and discovering snapshots never touches the cache.table_write.rs: a primary-key table withchangelog-producer=inputand manifest sidecars commits and reads its changelog incrementally; the changelog manifest and its sidecar are written and read through the target.rest_token_file_io.rs: options from the token and the catalog, and a table's own prefixes through the token FileIO.table/io_cache_routing_tests.rsdrives FileSystemCatalog tables over local OSS test servers: a Format Table whose file the origin replaced under a Paimon-shaped name, while the target kept the old copy, reads the new rows through the loaded table, a copy andTable::from_resolved_schema, and an Object Table reads its objects from the origin; a table whose schema alone setsdata-file.prefix, plus a copy with a dynamic prefix, abucket=-2primary-key table and a primary-key BLOB table write and read their data files, postpone files and managed packs through the target.Verified end to end
All three clients (Java, Python, Rust) created, wrote, read, time-travelled and dropped partitions of tables against local OSS mock endpoints that log every request, with one target, two targets by file type, write routing, and an OSS accelerator as origin with a read cache or OSS as the target. Every request a target saw was one these rules send to it. The pypaimon side (apache/paimon#10351) also ran its cases through a real OSS accelerator, as target and as origin.
API and Format
No public API or storage format change; routing is internal and configured through options.
Documentation
docs/src/getting-started.mddocuments the options under "OSS Cache Routing".