Dynamic Dense Retrieval (DDR)

Code for Dynamic Dense Retrieval: modular parameter-efficient adaptation with query-aware routing.

We freeze a shared dense-retrieval backbone, train a lightweight query-side LoRA per task, and keep modules in a growing bank. A training-free prototype router selects or composes the top-(K) specialists at inference time—without re-indexing documents.

Setup

Jobs are intended for the Slurm gpu partition (GH200) with conda env ir:

source slurm/ddr_env.sh

Or locally (after installing deps):

pip install -r requirements.txt
pip install -e .

Quick start (pilot)

bash slurm/submit_pilot.sh

This downloads SciFact / NFCorpus / FiQA, trains one LoRA per task, builds frozen indexes + routing prototypes, and evaluates:

Method	Description
frozen	Pretrained backbone only
oracle_lora	Correct task LoRA (upper bound for PEFT)
ddr_hard	Top-1 prototype routing
ddr_topk	Top-(K) module composition

Manual commands

python scripts/download_datasets.py --tasks scifact nfcorpus fiqa
python -m ddr.train --task scifact
python -m ddr.build_index --task scifact --mode both
python -m ddr.evaluate \
  --tasks scifact nfcorpus fiqa \
  --methods frozen oracle_lora ddr_hard ddr_topk

Full six-task suite

Supported tasks: fiqa, scifact, nfcorpus, hotpotqa, nq, msmarco.

Backbones used in the paper:

• Contriever (facebook/contriever)
• E5-base (intfloat/e5-base-v2) — scripts/run_e5_main.py
• Qwen3-Embedding-8B — scripts/run_qwen_main.py

Useful Slurm entry points under slurm/:

Script	Purpose
submit_full_stream.sh	Full Contriever train/eval stream
run_e5_main.sh	E5 Table-2 pipeline
run_qwen_train_eval.sh	Qwen3-8B pipeline

Layout

ddr/           # core library (train, index, router, evaluate)
scripts/       # experiment runners
slurm/         # cluster job scripts