feat: zvec embedded vector backend - #435
Closed
esafwan wants to merge 11 commits into
Closed
Conversation
Salvages the ZvecBackend from closed PR #168 (feature/zvec-vectorDB-integration) and adapts it to the knowledge-backend contract: - delete_chunks() returns the real count (filter-only query before delete) - get_stats() includes backend_type: zvec - get_advanced_config_schema() exposes HNSW/metric tuning knobs - injection-safe filter building (identifier whitelist + quote escaping, single '=' zvec filter DSL) - health_check(), supports_filters(), supports_hybrid_search() - guarded zvec import with a clear zvec>=0.2.1 install error Zvec is registered as a built-in backend (_BUILTIN_BACKENDS), wired into the Knowledge Source DocType (options, vector depends_on, VECTOR_KNOWLEDGE_TYPES, uses_vectors), the indexer embedding-config tuple, and pyproject dependencies. Includes mocked unit tests (26) mirroring test_redis_backend.py, plus registry tests covering built-in resolution and hook-override protection.
…idempotent zvec E2E cycle
The after_insert hook initialized backends with only chunk settings, so dimension-strict backends (zvec) got collections created with the 1536 fallback before the indexer could initialize with the real config.
Buffered upserts are not reflected in collection.stats until flushed, so get_stats reported chunk_count=0 right after indexing.
flush() inside get_stats can raise on a leftover collection LOCK and the exception handler then reports chunk_count=0. add_chunks already flushes after writing, so stats are current without it.
…script collection.stats is a pybind CollectionStats object whose repr looks like a dict; stats['doc_count'] raised TypeError which get_stats swallowed, reporting chunk_count=0. Add _doc_count() helper and use it in get_stats, delete_chunks and health_check. Update test mocks to match the real type.
esafwan
changed the title
Mirror the pgvector frontend parity for the zvec embedded backend: - add zvec to knowledgeTypes, VECTOR_KNOWLEDGE_TYPES, and labels - extend KnowledgeType union and zod knowledge_type enum with zvec - render the schema-driven AdvancedConfigFields card for zvec (no connection fields; tuning comes via advanced_config)
This was referenced Jul 26, 2026
…to feat/zvec-knowledge-backend # Conflicts: # frontend/src/components/knowledge/GeneralTab.tsx # frontend/src/components/knowledge/types.ts # frontend/src/data/knowledge.ts # frontend/src/types/knowledge.types.ts
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: zvec embedded vector backend
Adds zvec (Alibaba's in-process vector DB on Proxima, HNSW, cosine) as a built-in knowledge backend — the portable/no-server option alongside
sqlite_fts/sqlite_vec. Salvaged from #168's sound backend core and brought onto the current contract architecture; supersedes #149 and #168 (both closed: identical code, uninstallabletridz-zvecdep, 1056 commits behind, conflicting with the #280/#406 line).Why not the original PRs
zvecships cp313/cp314 wheels since 0.2.1 (0.6.0 now) — thetridz-zvecfork workaround is moot and was never published to PyPI.==) is a syntax error in zvec 0.6.0 (=required); its scalar fields also lacked the inverted indexes filtering needs.What's in this PR
ZvecBackend(KnowledgeBackend)(~480 lines, contract path 2 direct implementation): HUF-side embeddings, per-source collection in site private files, HNSW + configurable metric (cosine/l2/ip), injection-safe filter expressions (field whitelist + literal escaping), pre-countdelete_chunks, dimension validation on collection open,get_advanced_config_schema()(metric, HNSW m/ef),health_check,supports_filters._BUILTIN_BACKENDS+knowledge_typeoption + DocType/JS validation wiring +zvec>=0.2.1dependency.huf/ai/knowledge/hooks.py):on_knowledge_source_createdinitialized backends with only chunk settings — dimension-strict backends got stores created with the 1536 fallback before the indexer could initialize. Now uses_build_backend_config(doc).Testing (bench
16_kbreg, Frappe 16.27, Python 3.14, zvec 0.6.0)gemini-embedding-001(3072-dim) → index → semantic search (0.673) → stats/health → scoped deleteruff check/ruff formaton touched filesNotable zvec 0.6.0 behaviors found while testing:
collection.statsis a pybindCollectionStatsobject (dict-looking repr, not subscriptable — attribute access required); collections are single-writer with a LOCK file;delete_by_filterreturns None (hence pre-count);upsertbuffers untilflush().Dependencies
pyproject.toml:zvec>=0.2.1.