feat(knowledge): consolidate vector backends and registry - #467
Merged
esafwan merged 56 commits intoJul 26, 2026
Merged
Conversation
- embedding.py: configurable litellm_embedding_timeout (default 600s) passed as request_timeout; Ollama 400 single retry; batch_size=1 for ollama/ models - knowledge_source.py: test_connection whitelist API (tests pgvector + embedding) - knowledge_source.py: _warn_if_embedding_changed() warning when provider/model changes on a source with existing chunks - knowledge_source.js: Test Connection button handler with results dialog - knowledge_source.json: Test Connection button field (pgvector only) - knowledge_input.json: file field description updated to mention litellm_embedding_timeout
…sw params, distance_metric fix)
…+ backend contract doc
…drop dead sqlite_vec adapter
frappe.get_hooks() merges dict-valued hooks into a single dict of lists
({'redis': ['path.to.Class']}), not a list of dicts as _discover_backends()
assumed — so no hooked backend ever registered on a real site. Add
_iter_hook_backends() to normalise both shapes; first declaration of a
type key wins. Document the merge behaviour in BACKEND_CONTRACT.md and
update registry tests to mock the real hook shape.
…ds hook Supersedes #235. Direct KnowledgeBackend implementation on redisvl (SearchIndex + VectorQuery + Tag filters) instead of a LlamaIndex adapter: llama-index-vector-stores-redis 0.4.0 pins llama-index-core <0.13 and redisvl <0.4, and redisvl 0.3.x is broken on Python 3.14 (pydantic v1 shim), making the wrapper unusable on Frappe 16. - Registered through huf's own huf_knowledge_backends hook (dogfoods the registry); not a built-in - Connection settings via advanced_config schema (redis_host, redis_port, redis_username, redis_password, redis_index_prefix) per the contract - Site + source tag scoping on metadata, search filters and deletes - Embeddings always HUF-side; float32-byte vectors; cosine/flat index - 21 mocked unit tests + live E2E against Redis 8.8 (RediSearch)
Buffered upserts are not reflected in collection.stats until flushed, so get_stats reported chunk_count=0 right after indexing.
flush() inside get_stats can raise on a leftover collection LOCK and the exception handler then reports chunk_count=0. add_chunks already flushes after writing, so stats are current without it.
…script collection.stats is a pybind CollectionStats object whose repr looks like a dict; stats['doc_count'] raised TypeError which get_stats swallowed, reporting chunk_count=0. Add _doc_count() helper and use it in get_stats, delete_chunks and health_check. Update test mocks to match the real type.
Add Redis as a knowledge backend option alongside pgvector: - knowledgeTypes, VECTOR_KNOWLEDGE_TYPES and knowledgeTypeLabels - KnowledgeType union and zod form schema enum - Redis Connection Settings card in GeneralTab that renders the schema-driven AdvancedConfigFields (redis has no first-class DocType fields; connection settings come from the backend's get_advanced_config_schema)
Mirror the pgvector frontend parity for the zvec embedded backend: - add zvec to knowledgeTypes, VECTOR_KNOWLEDGE_TYPES, and labels - extend KnowledgeType union and zod knowledge_type enum with zvec - render the schema-driven AdvancedConfigFields card for zvec (no connection fields; tuning comes via advanced_config)
LlamaIndex-adapter backend (llama-index-vector-stores-weaviate) following the pgvector pattern: weaviate-client v4 connection (host/HTTP port/gRPC port/optional API key) via advanced config, per-source collection (Huf_<source>), site_name + knowledge_source metadata scoping for search/delete/clear/stats, optional hybrid (vector + BM25) search with configurable alpha, and mocked unit tests.
Built-in pinecone backend on LlamaIndexBackend via llama-index-vector-stores-pinecone (0.8.0, pinecone <8,>=7): - pinecone_backend.py: config validation (API key from advanced_config or PINECONE_API_KEY env, index-name rules, dimension, cloud/region), serverless index auto-creation with cosine metric, per-site/per-source namespace isolation plus site_name/knowledge_source metadata filters, delete_chunks via count-then-delete_nodes, clear via namespace purge, get_stats via describe_index_stats, advanced-config schema with plaintext-warning on the API key. - Registered as built-in: backend registry, Knowledge Source options and vector depends_on lists, VECTOR_KNOWLEDGE_TYPES, knowledge_source.js, indexer vector tuple, hooks.py comment, BACKEND_CONTRACT.md. - pyproject: llama-index-vector-stores-pinecone dependency. - 32 mocked unit tests (backend + registry), zvec-test style.
In-process, file-persisted vector backend on the LlamaIndex FAISS adapter (faiss.IndexFlatL2 / IndexIVFFlat), persisted under the site's private files at private/files/knowledge/<source>_faiss/. Salvages the private-files persistence idea from the closed PR #234 and adapts it to the knowledge-backend contract. FAISS has no metadata filtering and the adapter implements no delete, so supports_filters() is False, search relies on source-per-index isolation, and scoped delete_chunks rebuilds the index from the surviving chunks kept in a pickled sidecar (chunks.pkl) that maps FAISS positional ids to chunk text, metadata, and embeddings. Index and sidecar are persisted after every mutation. Registered as a built-in backend (registry, Knowledge Source options, vector-type lists, hooks comment, contract) with llama-index-vector- stores-faiss + faiss-cpu dependencies and 27 mocked unit tests.
…dge-backend # Conflicts: # huf/ai/knowledge/backends/BACKEND_CONTRACT.md # huf/ai/knowledge/backends/__init__.py # huf/ai/knowledge/indexer.py # huf/hooks.py # huf/huf/doctype/knowledge_source/knowledge_source.js # huf/huf/doctype/knowledge_source/knowledge_source.json # huf/huf/doctype/knowledge_source/knowledge_source.py
…dge-backend # Conflicts: # huf/ai/knowledge/backends/BACKEND_CONTRACT.md # huf/ai/knowledge/backends/__init__.py # huf/ai/knowledge/indexer.py # huf/hooks.py # huf/huf/doctype/knowledge_source/knowledge_source.js # huf/huf/doctype/knowledge_source/knowledge_source.json # huf/huf/doctype/knowledge_source/knowledge_source.py # pyproject.toml
…te is now a built-in); assert filters= kwarg
…to feat/zvec-knowledge-backend # Conflicts: # frontend/src/components/knowledge/GeneralTab.tsx # frontend/src/components/knowledge/types.ts # frontend/src/data/knowledge.ts # frontend/src/types/knowledge.types.ts
… into consolidation/vector-backends
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidated scope
Consolidates the complete contract-based vector backend chain.
Original PR chain
Merge order
PGVector -> registry -> Redis -> zvec -> Weaviate -> FAISS -> Pinecone. The Pinecone source branch already contained the full dependency chain and was used as the consolidation source.
Included functionality
Targets
integration/huf-consolidation-2026-07; included in final integration PR #468.