feat: FAISS embedded knowledge backend (revives #234 as contract-compliant built-in) - #437
Closed
esafwan wants to merge 3 commits into
Closed
feat: FAISS embedded knowledge backend (revives #234 as contract-compliant built-in)#437esafwan wants to merge 3 commits into
esafwan wants to merge 3 commits into
Conversation
In-process, file-persisted vector backend on the LlamaIndex FAISS adapter (faiss.IndexFlatL2 / IndexIVFFlat), persisted under the site's private files at private/files/knowledge/<source>_faiss/. Salvages the private-files persistence idea from the closed PR #234 and adapts it to the knowledge-backend contract. FAISS has no metadata filtering and the adapter implements no delete, so supports_filters() is False, search relies on source-per-index isolation, and scoped delete_chunks rebuilds the index from the surviving chunks kept in a pickled sidecar (chunks.pkl) that maps FAISS positional ids to chunk text, metadata, and embeddings. Index and sidecar are persisted after every mutation. Registered as a built-in backend (registry, Knowledge Source options, vector-type lists, hooks comment, contract) with llama-index-vector- stores-faiss + faiss-cpu dependencies and 27 mocked unit tests.
…dge-backend # Conflicts: # huf/ai/knowledge/backends/BACKEND_CONTRACT.md # huf/ai/knowledge/backends/__init__.py # huf/ai/knowledge/indexer.py # huf/hooks.py # huf/huf/doctype/knowledge_source/knowledge_source.js # huf/huf/doctype/knowledge_source/knowledge_source.json # huf/huf/doctype/knowledge_source/knowledge_source.py
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: FAISS knowledge backend
Adds FAISS as a built-in embedded knowledge backend via
llama-index-vector-stores-faiss+faiss-cpu(both compatible with llama-index-core 0.14.23 + Python 3.14). Revives the closed #234 on the current contract architecture.Design
FaissBackend(LlamaIndexBackend, KnowledgeBackend)(~450 lines): HUF-side embeddings, per-source index under siteprivate/files/knowledge/<source>_faiss/.chunks.pklsidecar (positional id → chunk data) — FAISS stores no text, so the sidecar powers search results and rebuilds.delete_chunks(input_id)finds matches via the sidecar, rebuilds the index from surviving embeddings (no re-embedding cost), re-keys, persists, returns the real count.supports_filters() -> False(search isolation is source-per-index; caller filters ignored with a debug log),supports_hybrid_search() -> False. Documented in docstring + schema help text.get_advanced_config_schema():faiss_index_type(flat/ivf),faiss_nlistwithvisible_whenIVF.Testing (bench
16_kbreg, Frappe 16.27, Python 3.14, faiss-cpu 1.14.3)gemini-embedding-001(3072-dim) → index → semantic search (0.605, L2-normalised) → stats (faiss_ntotal) → health → delete (rebuild) → 0ruff check/ruff formaton touched filesDependencies
pyproject.toml:llama-index-vector-stores-faiss,faiss-cpu.