Skip to content

Implement head_qa benchmark for inference evaluation pipeline - #46

Open
boysack wants to merge 9 commits into
epfLLM:mainfrom
boysack:feature/head_qa_benchmark
Open

Implement head_qa benchmark for inference evaluation pipeline#46
boysack wants to merge 9 commits into
epfLLM:mainfrom
boysack:feature/head_qa_benchmark

Conversation

@boysack

@boysack boysack commented May 29, 2026

Copy link
Copy Markdown

Overview

This PR officially integrates the head_qa (HEAlthcare Dataset) benchmark into the evaluation pipeline. This enables the evaluation of Meditron models against multi-choice questions derived from the Spanish healthcare system specialization exams.

Core Implementation

  • Benchmark Integration: Created the HeadQA class in benchmarks.py, enabling the downloading, parsing, and execution of the dataset within the standard inference.py and evaluate.py workflow.
  • Custom Preprocessing: Implemented the custom_preprocessing static method to accurately map head_qa's nested question/answer structure into the standard A/B/C/D/E multiple-choice format expected by the Meditron evaluator.
  • Dataset Routing Workaround: Bypassed current upstream Hugging Face deprecation blockers on the official repo by pointing the pipeline to a stable mirror (boysack/head_qa). This ensures the evaluation runs seamlessly out-of-the-box.
  • Native Configuration: Added meditron-7b and meditron-7b-awq Hugging Face paths directly into the master checkpoints dictionary in inference_pipeline.sh so the benchmark can be run natively without passing manual URIs.

Other fixes

  • vLLM & Memory Optimizations: Enabled AWQ quantized models (meditron-7B-AWQ), bypassed FlashInfer warmup spikes (enforce_eager=True), capped VRAM utilization at 0.8, and removed the hardcoded tensor_parallel_size.
  • vLLM & SamplingParams Deprecation: Removed deprecated parameters (best_of, use_beam_search).
  • Framework Maintenance: Updated Hugging Face caching path to namespace___resource (must be updated in the other benchmarks), patched the missing instructions.json template, and fixed evaluation output routing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant