Merge concurrent Eq/In filters into single In queries

[DZakh]

Optimize database queries by merging concurrent filters on the same field into fewer storage calls.

Summary

When multiple concurrent loadByFilter calls use Eq or In filters on the same field, they are now merged into a single In query. This reduces the number of database round-trips while maintaining correctness—loaded entities are matched against all registered indices, not just the merged query's filter.

Changes

• EntityFilter.res: Added merge function that collapses homogeneous filter batches:

  • Multiple Eq filters on the same field → single In filter
  • Multiple In filters on the same field → single In filter with combined values
  • Gt/Lt/And filters remain unchanged (no lossless single-query form without Or)

• LoadLayer.res: Updated loadByFilter to:

  • Register all filters' indices before querying
  • Merge filters before executing storage queries
  • Execute merged queries instead of individual ones

• Tests: Added comprehensive test coverage:

  • LoadLayer_test.res: Three new async tests verifying merge behavior for Eq, In, and non-mergeable Gt filters
  • EntityFilter_test.res: Unit tests for the merge function with various filter combinations

https://claude.ai/code/session_01RPHfND7Ge1qS598LWPvfif

Summary by CodeRabbit

• New Features

  • Optimized entity loading with automatic filter merging to reduce redundant database queries when multiple concurrent filter requests are issued.

• Tests

  • Added test coverage validating filter merging behavior for concurrent equality, inclusion, and comparison filter operations.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR optimizes concurrent data loading by introducing filter merging. A new EntityFilter.merge helper collapses homogeneous filter batches into fewer queries. LoadLayer.loadByFilter then uses this merge to pre-register indices and iterate over a superset query set instead of individual filters, reducing redundant storage operations for concurrent requests.

Changes

Query Merging Optimization

Layer / File(s)	Summary
EntityFilter merge contract and implementation packages/envio/src/db/EntityFilter.res, scenarios/test_codegen/test/EntityFilter_test.res	EntityFilter.merge reduces homogeneous filter batches by collapsing multiple Eq and In filters into single In filters with extracted/flattened values, leaving Gt, Lt, and And unchanged. Unit tests validate the merge behavior across filter type combinations.
LoadLayer integration with merged queries and concurrent tests packages/envio/src/LoadLayer.res, scenarios/test_codegen/test/LoadLayer_test.res	loadByFilter now pre-registers empty in-memory indices for all filters, merges the filter set, and iterates over merged queries for storage loads. Three concurrent test cases verify that Eq and In filters on the same field are merged into single DB queries while Gt filters remain separate.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• enviodev/hyperindex#1306: Both PRs modify packages/envio/src/LoadLayer.res's loadByFilter logic around how requested EntityFilter queries are loaded and empty in-memory indices are registered.
• enviodev/hyperindex#1303: Main PR extends the newly introduced EntityFilter with merge functionality and updates LoadLayer to aggregate filter queries before storage access, aligning with the shift to EntityFilter-driven predicates.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main optimization: merging concurrent Eq/In filters into single In queries for improved query efficiency.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}