Multimodal GraphRAG

A production-grade Retrieval-Augmented Generation system that combines a FAISS vector index, a Neo4j knowledge graph, and CLIP multimodal embeddings to answer questions over PDF documents and CSV datasets — including images.

Architecture

flowchart TD
    subgraph INGESTION
        A[PDF / CSV] --> B[PyMuPDF Extraction]
        B --> C[Text Chunks\n256 chars, 50 overlap]
        B --> D[PIL Images]
        C --> E[LLMGraphTransformer\nEntity + Relationship Extraction]
        D --> F[Gemini Vision\nImage Captioning]
        E --> G[(Neo4j Aura\nKnowledge Graph)]
        C --> H[CLIP ViT-B/32\nText Embeddings]
        D --> I[CLIP ViT-B/32\nImage Embeddings]
        F --> J[CLIP ViT-B/32\nCaption Embeddings]
        H & I & J --> K[(FAISS IndexFlatL2\nIn-memory + Disk)]
    end

    subgraph RETRIEVAL
        L[User Query] --> M[CLIP Text Embedding]
        M --> N[FAISS Top-k Search\nreturns chunk_ids]
        N --> O[fetch_chunk_subgraph\nNeo4j Cypher]
        N --> P[Raw Chunk Text]
        N --> Q[Image Captions]
    end

    subgraph GENERATION
        O & P & Q --> R[Hybrid Prompt\nText + Captions + Graph]
        R --> S[Gemini / Ollama LLM]
        S --> T[Answer]
    end

    K --> N
    G --> O

Loading

Why GraphRAG over Naive RAG?

Capability	Naive RAG	This System
Multi-hop reasoning across sections	✗	✓ via Neo4j graph traversal
Entity relationship context	✗	✓ LLMGraphTransformer
Image understanding	✗	✓ CLIP + Gemini vision captions
Shared text/image embedding space	✗	✓ CLIP ViT-B/32
Index survives restarts	✗	✓ FAISS saved to disk
Observable pipeline	✗	✓ JSON trace logs per query

Tech Stack

Component	Technology	Version
Vector search	FAISS (IndexFlatL2)	1.13+
Knowledge graph	Neo4j Aura	6.1+
Multimodal embeddings	CLIP ViT-B/32 (SentenceTransformers)	5.2+
PDF parsing	PyMuPDF (fitz)	1.27+
LLM / Graph extraction	Gemini 2.5 Flash / Ollama	langchain-google-genai 4.2+
Graph construction	LangChain LLMGraphTransformer	langchain-experimental 0.4+
Evaluation	RAGAS (faithfulness, relevancy, precision, recall)	0.2.6
UI	Gradio	6.6+

Pipeline Transparency

What gets chunked

• PDF: RecursiveCharacterTextSplitter(chunk_size=256, chunk_overlap=50) on raw page text. Each chunk gets a uuid4 ID. Images from the page are attached to all chunks from that page.
• CSV: Each row is atomic — one CLIP embedding per row, using selected node-column values.

How it's stored

• FAISS: IndexFlatL2 with a {faiss_pos → chunk_id} id_map. Text + image + caption vectors all point back to the same chunk_id. Saved to data/pdf_index/ after each build.
• Neo4j: Entities and relationships extracted per chunk, each tagged with source_chunk_id for traceability.
• Chunks: Serialized to data/pdf_index/chunks.json (text + captions, not PIL images).

What gets retrieved

1. Query → CLIP text embedding → FAISS top-3 → chunk_ids
2. chunk_ids → MATCH (n) WHERE n.source_chunk_id IN $ids OPTIONAL MATCH (n)-[r]-(m) RETURN n, r, m
3. Raw chunk text + image captions + graph triples → hybrid LLM prompt → answer

Setup

git clone https://github.com/your-username/Multimodal_GraphRAG.git
cd Multimodal_GraphRAG
python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt

Create a .env file:

GOOGLE_API_KEY=your_google_api_key
NEO4J_AURA_URI=neo4j+s://your-instance.databases.neo4j.io
NEO4J_AURA_USERNAME=neo4j
NEO4J_AURA_PASSWORD=your_password
NEO4J_AURA_DATABASE=neo4j
LLM_PROVIDER=gemini
GEMINI_MODEL=gemini-2.5-flash

Usage

PDF GraphRAG

python pdf_rag.py   # Opens at http://127.0.0.1:7860

1. Upload PDF — text, images, and graph are built automatically
2. Query — ask questions in natural language
3. Pipeline Trace tab — inspect exactly what was retrieved and sent to the LLM

CSV GraphRAG

python csv_rag.py   # Opens at http://127.0.0.1:7861

1. Upload CSV — choose node columns and define relationships
2. Build Graph — CLIP embeddings + Neo4j nodes/edges created
3. Query — natural-language search over rows + graph paths

Evaluation (Naive RAG vs GraphRAG)

# Generate testset and run full evaluation
python evaluate_pipeline.py --pdf dataset/testing_doc.pdf --testset-size 10

# Multi-hop questions (designed to stress-test graph advantage)
python evaluate_pipeline.py --multi-hop

# Fast direct QA (skip RAGAS graph generation)
python evaluate_pipeline.py --direct-qa

RAGAS metrics reported: faithfulness, answer_relevancy, context_precision, context_recall.

Project Structure

Multimodal_GraphRAG/
├── pdf_rag.py              # PDF pipeline Gradio app
├── csv_rag.py              # CSV pipeline Gradio app
├── evaluate_pipeline.py    # Evaluation CLI
├── src/
│   ├── core/
│   │   ├── config.py       # Settings from .env
│   │   ├── services.py     # Lazy-init Neo4j / LLM / embeddings
│   │   ├── embeddings.py   # CLIP multimodal embedding wrapper
│   │   ├── graph.py        # Cypher helpers (insert, fetch, merge)
│   │   └── trace.py        # Pipeline trace logger
│   ├── ingestion/
│   │   ├── pdf_ingester.py # Extract → chunk → caption → graph
│   │   └── csv_ingester.py # Load → embed rows → collect images
│   ├── retrieval/
│   │   └── vector_store.py # FaissIndex with save/load
│   ├── generation/
│   │   └── generator.py    # Prompt templates + LLM chains
│   └── evaluation/
│       ├── evaluator.py    # RAGAS test generation + scoring
│       ├── baselines.py    # Naive RAG (FAISS only)
│       └── graphrag_pipeline.py  # GraphRAG eval wrapper
├── data/                   # Auto-created: FAISS indexes + traces
└── dataset/                # PDFs and evaluation artifacts