Event-driven, multi-agent AI knowledge platform connecting GitHub, Slack, Jira, Notion, and Google Drive with hybrid vector search, LangGraph cyclic orchestration, and enterprise microservices.
- Executive Overview
- System Architecture
- Entity Relationship Diagram (ERD)
- Sequence Diagrams
- API Documentation
- Deployment Guide
- Observability & Monitoring
Knowledge Fabric AI (Codebase: Arch) is a production-grade enterprise knowledge platform designed to reduce knowledge search latency by 68% and increase semantic retrieval accuracy by 28%.
- Hybrid Search Engine: Combines dense vector similarity search (Qdrant HNSW) with sparse keyword search (PostgreSQL BM25) and Cross-Encoder re-ranking.
- Multi-Agent RAG Planner: Powered by LangGraph, executing cyclic state graphs across Retriever, Search, and Reasoning agents.
- Event-Driven Pipeline: Ingests real-time document pushes from GitHub, Slack, Jira, Notion, and Google Drive via Apache Kafka and AWS Lambda.
- High-Performance IPC: Microservices communicate over gRPC (HTTP/2 + Protobuf) with an Apollo Server GraphQL Gateway entry point.
Users
│
▼
Next.js + React Frontend
│
GraphQL Gateway (Apollo Server)
│
┌────────────────────┼────────────────────┐
│ │ │
Auth Service Search Service Chat Service
│ │ │
└───────────────┬────┴────────────────────┘
│
gRPC Communication
│
┌───────────────┬──────────┼──────────────┬──────────────┐
│ │ │ │ │
Document Service AI Orchestrator Analytics Notification User Service
(LangGraph)
│ │ │ │
└───────────────┴──────────┴──────────────┘
│
Apache Kafka
│
┌────────────┬───────────────┬──────────────┬──────────────┬──────────────┐
│ │ │ │ │
GitHub Slack Jira Notion Google Drive
Webhook Webhook Webhook Webhook Webhook
│ │ │ │ │
└────────────┴───────────────┴──────────────┴──────────────┘
AWS Lambda
│
Chunking + Embedding Generation
│
┌─────────────┴─────────────┐
│ │
Qdrant (Vector DB) PostgreSQL (AWS RDS)
│ │
└─────────────┬─────────────┘
│
Redis Cache
│
AWS S3 Storage
│
Docker → GitHub Actions → AWS EC2
│
Prometheus + Grafana
erDiagram
ORGANIZATIONS ||--|{ USERS : employs
ORGANIZATIONS ||--|{ WORKSPACES : owns
WORKSPACES ||--|{ DOCUMENTS : contains
WORKSPACES ||--|{ CHAT_SESSIONS : hosts
DOCUMENTS ||--|{ DOCUMENT_CHUNKS : splits_into
USERS ||--|{ CHAT_SESSIONS : initiates
CHAT_SESSIONS ||--|{ CHAT_MESSAGES : contains
ORGANIZATIONS {
uuid id PK
string name
string domain
string plan_tier
timestamp created_at
}
USERS {
uuid id PK
uuid organization_id FK
string email
string full_name
string role
string password_hash
}
WORKSPACES {
uuid id PK
uuid organization_id FK
string name
text description
}
DOCUMENTS {
uuid id PK
uuid workspace_id FK
string title
string source_type
string source_url
string s3_uri
int chunk_count
string status
}
DOCUMENT_CHUNKS {
uuid id PK
uuid document_id FK
int chunk_index
uuid qdrant_point_id
text content
int token_length
tsvector search_vector
}
CHAT_SESSIONS {
uuid id PK
uuid user_id FK
uuid workspace_id FK
string title
string agent_mode
}
CHAT_MESSAGES {
uuid id PK
uuid session_id FK
string sender
text content
float confidence_score
timestamp created_at
}
sequenceDiagram
autonumber
actor External as GitHub / Slack / Jira / GDrive
participant Webhook as Webhook Listener (8080)
participant Kafka as Apache Kafka Event Bus
participant Lambda as AWS Lambda Consumer
participant Qdrant as Qdrant Vector DB
participant RDS as PostgreSQL (RDS)
participant S3 as AWS S3 Storage
External->>Webhook: POST /api/webhooks/{source} (Payload event)
Webhook->>Kafka: Publish to 'webhook.events.raw'
Kafka-->>Lambda: Trigger Lambda Event Batch
Lambda->>S3: Fetch Raw File & Extract Text
Lambda->>Lambda: Execute Semantic Overlapping Window Chunking (512 tokens)
Lambda->>Lambda: Generate 1536-dim Vector Embeddings (OpenAI API)
Lambda->>Qdrant: Upsert Points (Dense Vectors + HNSW Index)
Lambda->>RDS: Insert Document Metadata & Full-Text Search tsvectors
Lambda-->>Kafka: Publish 'document.indexing.completed' event
sequenceDiagram
autonumber
actor User as React / Next.js Client
participant Gateway as GraphQL Gateway (Apollo Server)
participant Orchestrator as AI Orchestrator (LangGraph)
participant Search as Search Service (gRPC)
participant Qdrant as Qdrant (Vector DB)
participant RDS as PostgreSQL (RDS)
participant LLM as OpenAI LLM Model
User->>Gateway: GraphQL Mutation executeAgentWorkflow(prompt)
Gateway->>Orchestrator: gRPC ExecuteLangGraphWorkflow()
Orchestrator->>Orchestrator: Node 1: LangGraph Planner (Decompose intent)
par Parallel Multi-Agent Retrieval
Orchestrator->>Search: Node 2: Retriever Agent (Qdrant Dense Vector)
Search->>Qdrant: Cosine Similarity Search (k=10)
Qdrant-->>Search: Return Dense Matches
and
Orchestrator->>Search: Node 3: Search Agent (Postgres Sparse BM25)
Search->>RDS: SQL SELECT ts_rank_cd() Search
RDS-->>Search: Return Sparse Matches
end
Orchestrator->>Orchestrator: Node 4: Reasoning Agent (Verify Context)
Orchestrator->>Orchestrator: Node 5: Hybrid Reranker (Cross-Encoder score sorting)
Orchestrator->>LLM: Node 6: LLM Response & Citation Synthesis
LLM-->>Orchestrator: Streamed Answer Tokens + Citation Metadata
Orchestrator-->>Gateway: gRPC Stream WorkflowResponse
Gateway-->>User: GraphQL Response Stream
type Query {
me: User!
hybridSearch(query: String!, workspaceId: String!, topK: Int): SearchResult!
}
type Mutation {
executeAgentWorkflow(
sessionId: String!
prompt: String!
workspaceId: String!
agentMode: String
): ChatResponseChunk!
}
type ChatResponseChunk {
nodeId: String!
stepType: String!
tokenChunk: String!
confidenceScore: Float!
isFinal: Boolean!
}syntax = "proto3";
package enterprise.ai;
service AIOrchestrator {
rpc ExecuteLangGraphWorkflow (WorkflowRequest) returns (stream WorkflowResponse);
}
service SearchService {
rpc HybridSearch (SearchRequest) returns (SearchResponse);
}- Docker 24.0+ & Docker Compose 2.20+
- Node.js 20.x+ & npm 10.x+
- Python 3.11+
-
Clone the repository:
git clone https://github.com/enterprise/knowledge-fabric-ai.git cd knowledge-fabric-ai -
Start all services with Docker Compose:
docker-compose up -d --build
-
Verify running containers:
docker-compose ps
-
Access Applications:
- Interactive Visualizer & Simulator: Open
index.htmlin browser - Apollo GraphQL Gateway:
http://localhost:4000/graphql - Prometheus Metrics:
http://localhost:9090 - Grafana Dashboard:
http://localhost:3000(User:admin, Pass:admin)
- Interactive Visualizer & Simulator: Open
- Prometheus: Scrapes metrics every 15s from GraphQL Gateway, gRPC services, Qdrant, and Redis.
- Grafana: Pre-configured dashboards tracking:
p99 Vector Search Latency(< 5.8 ms)Kafka Consumer LaggRPC RPC Handling Bucket DurationRedis Cache Hit Ratio(> 94%)