Simplify InMemoryTable by removing generic wrapper - #1265
Merged
Conversation
…nd indices The generic InMemoryTable.t<'key, 'val> (dict + hash function) was overkill for raw events and entity indices. Raw events now use a plain dict keyed by the event id string, and the index maps use plain dicts keyed by the index's serialized string. Removes the unused addIdToIndex helper.
Raw events have no in-memory dedup requirement: each event is processed once per batch and the raw_events table uses a serial primary key, so a plain array replaces the dict (the key was only ever values()'d at write time). Drops the now-unused getEventIdKeyString helper. Also iterate index values via Utils.Dict.forEach instead of materializing an intermediate array.
Contributor
|
Warning Review limit reached
More reviews will be available in 16 minutes and 39 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR refactors internal storage mechanisms throughout the event processing pipeline, replacing a generic keyed-table wrapper with direct data structures: raw events migrate from a keyed table to arrays, and entity index metadata migrate from a keyed-table wrapper to nested dicts. ChangesStorage Data Structure Refactoring
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/envio/src/InMemoryTable.res (1)
187-195: ⚡ Quick winCentralize index key serialization.
Both lookup paths rebuild the storage key by hand even though this table is keyed by
TableIndices.Index.toString. That duplication will silently breakhasIndex/getUnsafeOnIndexif the serializer ever changes. Please funnel these lookups through one shared helper, ideally the same serializer used on insert.♻️ Suggested cleanup
+ let serializeIndexLookupKey = (~fieldName, ~operator, ~fieldValueHash) => + `${fieldName}:${(operator :> string)}:${fieldValueHash}` + let hasIndex = (inMemTable: t, ~fieldName, ~operator: TableIndices.Operator.t) => fieldValueHash => { switch inMemTable.fieldNameIndices->Utils.Dict.dangerouslyGetNonOption(fieldName) { | None => false | Some(indicesSerializedToValue) => { - let key = `${fieldName}:${(operator :> string)}:${fieldValueHash}` + let key = serializeIndexLookupKey(~fieldName, ~operator, ~fieldValueHash) indicesSerializedToValue->Utils.Dict.dangerouslyGetNonOption(key) !== None } } } let getUnsafeOnIndex = (inMemTable: t, ~fieldName, ~operator: TableIndices.Operator.t) => { let getEntity = inMemTable->getUnsafe fieldValueHash => { switch inMemTable.fieldNameIndices->Utils.Dict.dangerouslyGetNonOption(fieldName) { | None => JsError.throwWithMessage(`Unexpected error. Must have an index on field ${fieldName}`) | Some(indicesSerializedToValue) => { - let key = `${fieldName}:${(operator :> string)}:${fieldValueHash}` + let key = serializeIndexLookupKey(~fieldName, ~operator, ~fieldValueHash) switch indicesSerializedToValue->Utils.Dict.dangerouslyGetNonOption(key) {Also applies to: 199-208
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/InMemoryTable.res` around lines 187 - 195, The index lookup builds the storage key inline in hasIndex (and similarly in getUnsafeOnIndex), duplicating TableIndices.Index.toString logic; extract a single helper (e.g., serializeIndexKey or TableIndices.Index.toKey) and replace the inline template `${fieldName}:${operator}:${fieldValueHash}` with a call to that helper everywhere keys are created/used (including insert code that currently uses TableIndices.Index.toString), so all lookups/inserts use the same serializer and avoid drift if serialization changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/envio/src/EventProcessing.res`:
- Line 96: The current append-only write
(inMemoryStore.rawEvents->Array.push(rawEvent)) can enqueue duplicates because
rawEvents is only cleared after InMemoryStore.writeBatch; add idempotent
buffering by tracking a companion key set (e.g., rawEventsKeys or rawEventIndex)
keyed by the per-(chainId,eventId) dedupe key and check it before pushing:
compute the unique key from rawEvent, skip push if key exists, otherwise add key
to the set and push the rawEvent; ensure the set is cleared alongside rawEvents
in the same place where InMemoryStore.writeBatch currently clears rawEvents so
replayed batches won't reintroduce duplicates.
---
Nitpick comments:
In `@packages/envio/src/InMemoryTable.res`:
- Around line 187-195: The index lookup builds the storage key inline in
hasIndex (and similarly in getUnsafeOnIndex), duplicating
TableIndices.Index.toString logic; extract a single helper (e.g.,
serializeIndexKey or TableIndices.Index.toKey) and replace the inline template
`${fieldName}:${operator}:${fieldValueHash}` with a call to that helper
everywhere keys are created/used (including insert code that currently uses
TableIndices.Index.toString), so all lookups/inserts use the same serializer and
avoid drift if serialization changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3a8300e0-7b44-4a63-8645-b047fe6d5c09
📒 Files selected for processing (4)
packages/envio/src/EventProcessing.respackages/envio/src/EventUtils.respackages/envio/src/InMemoryStore.respackages/envio/src/InMemoryTable.res
💤 Files with no reviewable changes (1)
- packages/envio/src/EventUtils.res
hasIndex/getUnsafeOnIndex rebuilt the lookup key inline, duplicating SingleIndex.toString's format. Funnel both insert and lookup through Index.toStringByParts so the format has a single source of truth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the generic
InMemoryTable.twrapper type and replaces all usages with directdictoperations. This simplifies the codebase by eliminating an unnecessary abstraction layer that only wrapped dictionary operations with a hash function.Key Changes
t<'key, 'val>type and all associated wrapper functions (make,set,get,setByHash,hasByHash,getUnsafeByHash,values)dictdirectly instead oft<...>:indicesSerializedToValue: nowdict<indexWithRelatedIds>(keyed byTableIndices.Index.toString)indexFieldNameToIndices: nowdict<indicesSerializedToValue>(keyed byTableIndices.Index.getFieldName)Dictoperations and explicit hash key generationrawEventsstorage fromInMemoryTable.t<rawEventsKey, InternalTable.RawEvents.t>toarray<InternalTable.RawEvents.t>, eliminating the need for thehashRawEventsKeyfunctiongetEventIdKeyStringfunctionsetmethodImplementation Details
index->TableIndices.Index.toStringfor dictionary keys)rawEventsfield now uses a simple array instead of a keyed dictionary, as events are processed sequentially and don't require lookup by keyUtils.Dict.dangerouslyGetNonOptionwith explicit key generationhttps://claude.ai/code/session_0189AsEpZmVSjhrox3K4PKEc
Summary by CodeRabbit