feat: Add health monitoring and fallback system

[esafwan]

Summary

Implements health monitoring and automatic fallback system for vector backends.

Features

• BackendHealthMonitor - Health check caching (5-min TTL), status tracking
• FallbackRetriever - Automatic failover to SQLite on backend failure
• Scheduled job - Background health checks every 5 minutes

Safety

• Graceful degradation to SQLite
• Health status visible in DocType
• Configurable fallback per knowledge source

Files

• huf/ai/knowledge/health_monitor.py
• huf/ai/knowledge/fallback_retriever.py
• huf/huf/scheduled_job_type/health_check_backends/

Related

Ensures reliability when using external vector databases.

[esafwan]

TODO before this PR is taken up: rebuild on the PR #280 contract

Since this PR was opened, HUF established, in PR #280 (in review), the KnowledgeBackend contract and get_advanced_config_schema(). Health monitoring and fallback are still valid future features, but they should be expressed as capabilities on the backend contract (audit P2 item 9), not as a parallel BackendHealthMonitor + FallbackRetriever system that duplicates backend initialization and bypasses the standard retriever.

This PR also imports BackendFactory from factory.py in the new pgvector_backend.py, which is not the chosen registry design.

• Remove huf/ai/knowledge/backends/factory.py usage from pgvector_backend.py; register pgvector via get_backend() in backends/__init__.py instead.
• Refactor BackendHealthMonitor so it delegates to a health_check() method on KnowledgeBackend rather than maintaining its own backend lifecycle; cache results via Frappe cache if needed.
• Express supports_filters / supports_hybrid_search as contract capability flags and make fallback_retriever.py consult them before applying filters or hybrid logic.
• Wire fallback into huf/ai/knowledge/retriever.py or keep it as an opt-in wrapper in huf/ai/knowledge/tool.py; avoid forcing fallback on every knowledge search.
• Remove or make optional the scheduled health-check job in huf/huf/scheduled_job_type/health_check_backends/ until the feature is proven; ensure it does not spam logs on SQLite-only sites.
• Rebase onto post-feat: add PGVector knowledge source #280 code and pass the live e2e test bar (UI save → index → query with primary and simulated fallback).

See the cross-backend audit (P2 item 9) and pgvector_backend.py / chroma_backend.py (PR #280) as reference implementations.

[esafwan]

Closing as stale / re-scoped.

The knowledge-backend architecture has standardised on the #280 → #406 → #434 line: LlamaIndexBackend base class (#280), the huf_knowledge_backends hook registry + BACKEND_CONTRACT.md (#406), and hook-registered backends (#434). This PR predates that line: it targets the hardcoded get_backend() registry, carries the ~5k-line PRD/planning doc bundle, and conflicts with the contract's config model (advanced_config schemas vs separate settings DocTypes).

A backend from this PR can be revived as a small contract-compliant implementation (see huf/ai/knowledge/backends/BACKEND_CONTRACT.md on develop after #280/#406 merge): subclass LlamaIndexBackend (~50–100 lines) or implement KnowledgeBackend directly, register via huf_knowledge_backends in the owning app's hooks.py, expose tuning via get_advanced_config_schema(), and pass the live-test bar. The Redis backend in #434 is the reference implementation.