feat: tiered data storage#7
Open
enigbe wants to merge 4 commits into
Open
Conversation
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
9 times, most recently
from
29f47f3 to
264aa7f
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
a30cbfb to
1e7bdbc
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
b5e980f to
67d47c2
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
95285b0 to
4b2d345
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
cba29a3 to
db1fe83
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
db1fe83 to
35f9ec2
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
e89ada5 to
3abd0a5
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
8f122cb to
7155c40
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
7720935 to
181db03
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
5da76a8 to
dec4aa7
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
c1563e3 to
a2458e4
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
8dbb312 to
0ae61b7
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
7 times, most recently
from
cadf81f to
1f2cd4d
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
3eef01d to
c4aa2b0
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
0a240d4 to
10b9721
Compare
This commit adds `TierStore`, a tiered `KVStore` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary+backup. Reads and lists do not consult the backup store during normal operation. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals
Add native builder support for configuring ephemeral storage and a local SQLite backup mirror. Wrap the primary store in TierStore during node construction and create configured secondary stores using dedicated SQLite database files. Implement paginated listing through TierStore and update filesystem-backed tests to use FilesystemStoreV2. Add full-cycle integration coverage verifying durable backup mirroring.
Preserve call-time ordering for ephemeral writes and removes by routing them through the same versioned lock path as primary-backed mutations. Add regression coverage for stale ephemeral writes and removes.
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
7750cf8 to
9820126
Compare
Previously, the implementations for `list` and paginated listing on `TierStore`
contained a bug where, if the ephemeral store was configured, these operations
could return only keys from the ephemeral tier for shared root namespaces such
as ("", ""). This meant durable keys saved to the primary store could be ignored
whenever their namespace was shared with ephemeral-cached keys.
While the solution to `list` seemed straightforward, the one for paginated
listing raised questions about the potential difference in store types between
the primary and ephemeral stores, and the conflict that would arise from them
having independent pagination orders and tokens. `TierStore`, as currently
designed, does not maintain a cross-store cursor, so we rely on a simple overlay
strategy: paginate through primary first, then append ephemeral keys once primary
is on its terminal page.
This does mean the resulting paginated listing is not a strict cross-store
creation-order merge: ephemeral keys are returned after primary pagination is
exhausted, regardless of their creation order in the ephemeral store. We accept
that tradeoff because `TierStore` has no shared creation order across backing
stores, and the overlay is bounded to the current ephemeral-cached key set.
This commit allows us to merge listings across `TierStore`'s primary and
ephemeral tiers by treating the latter as an overlay for ephemeral-cached keys
only. We filter stale primary copies of ephemeral-cached keys, append live
ephemeral copies once primary pagination is complete, and only consult the
ephemeral store for namespaces that can contain ephemeral-cached keys.
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
9820126 to
81e65cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
We introduce
TierStore, aKVStoreimplementation that manages data acrossthree distinct storage layers.
The layers are:
(e.g., network graph). This tier aims to improve latency by leveraging a
local
KVStoredesigned for fast/local access.asynchronously/lazily to avoid blocking primary store operations.
We also permit the configuration of
Nodewith these stores allowingcallers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the
Nodewith TierStore's primary store. These configurationoptions also extend to our foreign interface, allowing bindings target to build the
Node with their own
ffi::KVStoreimplementations.A sample Python implementation is added and tested.
Additionally, we add comprehensive testing for
TierStoreby introducingTierStorecore functionality.Nodebuilt with tiered storage.ffi::KVStoreimplementations.Concerns
It is worth considering the way retry logic is handled, especially because of nested
retries.
TierStorecomes with a basic one by default but there areKVStoreimplementationsthat come with them baked-in (e.g.
VssStore), and thus would have no need forthe wrapper-store's own logic.