Visual RAG on Refined ChartQA Dataset

This repository refines the ChartQA dataset and implements a Visual Retrieval-Augmented Generation (Visual RAG) pipeline for multimodal chart-based reasoning.

---

ChartQA Refinement for RAG

Overview

data/refine_chartqa.py refines ChartQA questions to make them suitable for Retrieval-Augmented Generation (RAG) tasks.
Original ChartQA queries often depend on the chart image (e.g., “What is the unfavourable value in 2014?”).
Using a Vision-Language Model (VLM) such as Qwen/Qwen2.5-VL-32B-Instruct, the script rewrites them into standalone questions that preserve intent while being well-defined without the reference image (e.g., "According to a survey tracking views on the Ennahda party over several years, what was the percentage of unfavorable views reported in 2014?").

Each refined entry includes:

{
  "refined_query": "Refined standalone question",
  "refined_label": ["Answer"]
}

Note: Even though the prompt instructs the model to maintain the same intent and answer, mismatches can occur, especially for smaller, less-capable checkpoints. To ensure dataset quality, entries with a refined_label differing from the original label are dropped before integration into the RAG pipeline.

Usage

Refer to refine_chartqa_qas.ipynb for end-to-end refinement experiments using different generator checkpoints: Qwen/Qwen2.5-VL-7B-Instruct, Qwen/Qwen2.5-VL-32B-Instruct, and Qwen/Qwen2.5-VL-72B-Instruct-AWQ.

---

Multimodal RAG

Overview

vrag/visual_rag.py implements a Visual Retrieval-Augmented Generation (Visual RAG) framework for chart-based multimodal reasoning.
It combines retrieval over chart images and visual-language generation to answer refined ChartQA-style queries using only visual evidence.

Pipeline

1. Retrieve: Fetch top–k relevant charts for a query using a visual retriever (ColPaliRetriever, LlamaNemoRetriever).
2. Generate: Produce an answer with cited chart(s) using a VLM (QwenVLGenerator).
3. Evaluate: Measure results with various metrics such as precision, recall, soft match, and correctness (using QwenVLGenerator as a judge).

Usage

Refer to the following notebooks for pipeline runs and benchmarking results:

• visual_rag_bench.ipynb: Visual RAG using vidore/colpali-v1.3 / vidore/colqwen2-v1.0 retrievers and Qwen/Qwen2.5-VL-7B-Instruct / Qwen/Qwen2.5-VL-32B-Instruct as generator.
• visual_rag_bench2.ipynb: Visual RAG using nvidia/llama-nemoretriever-colembed-3b-v1 retriever and Qwen/Qwen2.5-VL-7B-Instruct / Qwen/Qwen2.5-VL-32B-Instruct as generator.

---

Environment Setup

Load the Python module and CUDA:

module load python/3.10.12
module load cuda-12.4

Setup the environment:

python -m venv .venv
source .venv/bin/activate

Install dependencies:

1. To run the RAG pipeline with ColPali/ColQwen retriever:

pip install --no-deps -r requirements.txt
pip install flash-attn==2.7.3 --no-build-isolation

2. To run the RAG pipeline with LlamaNemo retriever:

pip install --no-deps -r requirements2.txt
pip install flash-attn==2.7.3 --no-build-isolation

(Note that LlamaNemo retriever requires transformers==4.49.0 which conflicts byaldi dependencies required for ColPali/ColQwen retriever.)

---

Results

The following metrics are used to evaluate the RAG pipeline:

Metric	Type	Description
ret_precision@3	Retrieval	Proportion of retrieved charts that match the ground-truth chart. Measures retrieval accuracy.
ret_recall@3	Retrieval	Proportion of relevant charts that were successfully retrieved. Measures retrieval completeness.
gen_precision@3	Generation Citation(s)	Proportion of charts referenced by the model that match the ground-truth chart(s). Evaluates reference correctness in reasoning.
gen_recall@3	Generation Citation(s)	Proportion of ground-truth chart(s) correctly referenced by the model. Indicates how well the model identifies the correct visual evidence.
gen_exact_match	Generation Short Answer	Binary score (1 or 0) indicating whether the model’s short answer exactly matches the ground-truth answer (case-insensitive).
gen_soft_match	Generation Short Answer	Fuzzy string similarity between the model’s short answer and the ground-truth answer (normalized between 0–1).
gen_soft_match_partial	Generation Short Answer	Partial fuzzy match allowing substring overlap; useful when the model’s short answer partially matches the ground truth (e.g., “12” vs. “12.0”).
gen_correctness	Generation Full Answer (LLM-as-Judge)	Binary correctness (1 or 0) scored by an LLM evaluator checking if the model’s full reasoning-based answer aligns with the ground truth with respect to the query. Reflects semantic correctness beyond formatting.

Visual RAG Results

RAG results on 459 (out of 500) QAs, refined using the Qwen2.5-VL-32B-Instruct model:

RAG Setup	ret_precision	ret_recall	gen_correctness	gen_exact_match	gen_soft_match	gen_soft_match_partial	gen_precision	gen_recall
RAG (colpali-v1.3, Qwen2.5-VL-7B-Instruct)	32.53	97.60	73.42	48.15	71.30	78.19	94.52	95.86
RAG (colpali-v1.3, Qwen2.5-VL-32B-Instruct)	32.53	97.60	82.79	53.16	77.14	84.98	93.46	93.90
RAG (colqwen2-v1.0, Qwen2.5-VL-7B-Instruct)	32.39	97.17	73.64	48.58	71.58	78.65	94.92	96.08
RAG (colqwen2-v1.0, Qwen2.5-VL-32B-Instruct)	32.39	97.17	84.10	53.38	77.23	84.73	93.06	93.68
RAG (llama-nemoretriever-colembed-3b-v1, Qwen2.5-VL-7B-Instruct)	32.39	97.17	72.11	47.93	70.98	78.20	94.30	95.64
RAG (llama-nemoretriever-colembed-3b-v1, Qwen2.5-VL-32B-Instruct)	32.39	97.17	83.66	51.63	75.08	82.81	91.79	92.37

Where the LLM-as-Judge evaluator is Qwen2.5-VL-32B-Instruct.

ChartQA Refinement Ablation

RAG results on 100 refined QAs using different refinement models:

Setup	ret_precision	ret_recall	gen_correctness	gen_exact_match	gen_soft_match	gen_soft_match_partial	gen_precision	gen_recall
72 Refined QAs (Qwen2.5-VL-7B-Instruct)	29.63	88.89	62.50	43.06	67.56	76.10	87.96	88.89
86 Refined QAs (Qwen2.5-VL-32B-Instruct)	31.78	95.35	75.58	45.35	69.36	76.52	91.47	95.35
95 Refined QAs (Qwen2.5-VL-72B-Instruct-AWQ)	32.98	98.95	67.37	41.05	67.26	75.68	94.04	98.95

Where the RAG setup is (colpali-v1.3, Qwen2.5-VL-7B-Instruct) and LLM-as-judge evaluator is Qwen2.5-VL-7B-Instruct.

---

Citation

If you use this work or build upon it, please cite:

@misc{chartqa-rag,
  author = {Saeed Abbasi and Will Huang and Suzy Zhang},
  title = {ChartQA Refinement and Visual Retrieval-Augmented Generation},
  year = {2025},
  url = {https://github.com/saeedabc/chartqa_visual_rag}
}

Licence

This project is licensed under the MIT License.

Acknowledgement

We thank the Vector Institute for providing compute resources and facilitation as part of the Multimodal Bootcamp.