Feature/implement zvec portable vectordb

[esafwan]

Summary

Adds Zvec as a portable, in-process vector database backend for the Huf Knowledge System, enabling semantic (vector similarity) search alongside the existing SQLite FTS keyword search.

---

Background: What is Zvec?

Zvec is a lightweight, in-process vector database built on Alibaba's Proxima approximate nearest-neighbor engine. It stores embeddings as portable .zvec collection files — no external server process required.

Why Zvec for Huf?

Requirement	Zvec
In-process (no separate server)	✅
Portable file-based storage	✅ .zvec files in /private/files/knowledge/
HNSW index with cosine similarity	✅
Schema-driven (typed scalar + vector fields)	✅
Multi-site / multi-agent safe	✅ (read-concurrent, write-serialized via Redis locks)

This mirrors the existing SQLite FTS pattern: one portable file per Knowledge Source, stored in Frappe's private files directory, shareable across agents and sites.

---

What Was Added

New Files

File	Purpose
huf/ai/knowledge/embedding.py	Model-agnostic embedding generation via LiteLLM (get_embedding, get_embeddings, resolve_embedding_config)
huf/ai/knowledge/backends/zvec_backend.py	ZvecBackend(KnowledgeBackend) — full CRUD + vector search
huf/ai/knowledge/backends/zvec_llamaindex.py	Optional ZvecVectorStore(BasePydanticVectorStore) adapter for LlamaIndex pipeline compatibility

Modified Files

File	Change
huf/ai/knowledge/backends/__init__.py	Register zvec in get_backend() factory
huf/huf/doctype/knowledge_source/knowledge_source.json	Add zvec to knowledge_type options; add Vector Settings section (embedding_model, vector_dimension, embedding_provider)
huf/huf/doctype/knowledge_source/knowledge_source.py	Add validate_zvec_settings()
huf/huf/doctype/knowledge_source/knowledge_source.js	Remove hard-coded sqlite_fts-only restriction; dynamic field visibility for vector settings
huf/ai/knowledge/indexer.py	_build_backend_config() helper — passes embedding config to vector backends
pyproject.toml	Add zvec dependency

---

Architecture

Knowledge Source (DocType)
  ├── knowledge_type: "sqlite_fts" | "zvec"
  ├── embedding_model (zvec only)
  ├── vector_dimension (zvec only)
  └── embedding_provider (zvec only)

Indexing Pipeline:
  Knowledge Input → TextExtractor → chunk_text()
    → _build_backend_config(source)
    → backend.initialize(source, config)
    → backend.add_chunks(chunks)  # embeds + upserts

Search Pipeline:
  query → backend.search(query, top_k)
    → embed query → zvec.VectorQuery → collection.query()
    → List[ChunkResult]

The embedding infrastructure is provider-agnostic — it uses litellm.embedding() and reads model/API key config from the Knowledge Source and its linked AI Provider document.

---

How to Test

Prerequisites

1. An AI Provider with an embedding-capable model configured (e.g., OpenAI text-embedding-3-small, Gemini models/embedding-001)
2. bench pip install zvec (or bench setup requirements to install from updated pyproject.toml)

Steps

1. Create a Knowledge Source

   • Go to Knowledge Source list → New
   • Set Knowledge Type = zvec
   • The "Vector Settings" section should appear
   • Set Embedding Model (e.g., text-embedding-3-small)
   • Set Vector Dimension (e.g., 1536 for OpenAI, 768 for Gemini)
   • Optionally link an Embedding Provider
   • Save — validation should pass

2. Validate field visibility

   • Switch Knowledge Type back to sqlite_fts — vector settings should hide
   • Switch to zvec — vector settings should reappear and become required

3. Add a Knowledge Input

   • Add text or file input to the Knowledge Source
   • Trigger indexing (process the input)
   • Check that a .zvec file is created in {site}/private/files/knowledge/

4. Test search

   • From bench console or via the retriever, call:

     from huf.ai.knowledge.backends import get_backend
     backend_class = get_backend("zvec")
     backend = backend_class()
     backend.initialize("YOUR-SOURCE-NAME", {
         "embedding_model": "text-embedding-3-small",
         "vector_dimension": 1536,
     })
     results = backend.search("your query here", top_k=3)
     for r in results:
         print(r.score, r.text[:100])

5. Verify existing FTS still works — creating/using sqlite_fts Knowledge Sources should be unaffected

---

Commit History

77f1a45 feat: add embedding infrastructure for vector backends
fe1c900 feat: implement ZvecBackend for portable vector search
5da85b6 feat: add LlamaIndex VectorStore adapter for Zvec
006155e feat: add zvec to Knowledge Source DocType and backend registry
a437b87 feat: pass embedding config from indexer to vector backends
2d77d9b build: add zvec dependency to pyproject.toml

[esafwan]

TODO before this PR is taken up: align with the HUF knowledge-backend standard

Since this PR was opened, HUF adopted a cross-backend contract in PR #280 (pgvector + generic advanced_config schema). The diff in huf/ai/knowledge/backends/zvec_backend.py should be treated as a reference implementation, not a merge candidate.

Zvec has no LlamaIndex integration, so under the new standard it must either justify itself under the portable-exception clause (shareable local file, zero external services) or be dropped in favor of sqlite_vec, which already covers that niche. Git history already shows zvec was added and then removed once.

• Decide exception-vs-drop and add a short rationale in the PR description: what does zvec offer over sqlite_vec (file size, recall, query latency, etc.)?
• If kept: make ZvecBackend conform to KnowledgeBackend in backends/__init__.py and implement get_advanced_config_schema() for tuning knobs (e.g. index type, distance metric, query-time behavior). Hand-rolled is acceptable under the portable exception.
• If kept: drop or productize huf/ai/knowledge/backends/zvec_llamaindex.py — do not leave an orphaned adapter.
• Move any backend-specific connection/persistence tuning out of hardcoded initialize() in zvec_backend.py and into advanced_config or first-class Knowledge Source field groups following the pgvector_*/chroma_* pattern in PR feat: add PGVector knowledge source #280.
• Confirm embeddings stay HUF-side (huf.ai.knowledge.embedding) and are passed into the collection; do not let zvec generate embeddings.
• Run a live end-to-end test (UI save → index → query → verify .zvec artifact) matching the bar set in PR feat: add PGVector knowledge source #280.

See the cross-backend audit (A3 / P2 reconcile historical vector backends) and use pgvector_backend.py / chroma_backend.py (PR #280) as reference implementations.

[esafwan]

Closing as duplicate of #168 — every shared file is byte-identical (same git blobs), and #168 is the better-documented iteration. See #168 for the zvec discussion; a fresh contract-compliant zvec PR is being prepared to supersede both.