Repository navigation
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved moderate issues affect processor selection, fast-mode execution, image handling, and downstream modality preservation.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (3)
What changed in this PR
Adds an opt-in pymupdf4llm PDF processor for structured Markdown extraction in fast mode.
Changes:
- Adds Markdown, paragraph metadata, and image extraction.
- Preserves Markdown blank lines.
- Adds the dependency, configuration documentation, and processor-selection tests.
| File | Summary | Review notes |
|---|---|---|
tests/test_processor_selection.py |
Tests PDF processor selection. | — |
src/mmore/process/utils.py |
Adds optional blank-line preservation. | — |
src/mmore/process/processors/pdf_pymupdf4llm_processor.py |
Implements the new Markdown-based PDF processor. | Six findings: five moderate (votes: 1, 1, 2, 1, 2) and one nit (4 votes), covering image-link matching, default registration priority, missing extraction-path tests, reused image XObjects, modality chunking, and device-specific fast-mode handling. |
pyproject.toml |
Adds the pymupdf4llm dependency. |
— |
examples/process/config.yaml |
Documents PDF processor choices. | — |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
# Conflicts: # uv.lock
…on when processing pdf
|
As discussed with @fabnemEPFL, pymupdf4llm will be used as the default processor for the Index API. The current behaviour is that a processor exposes a standard processor and potentially a "fast" processor, with the intention that the fast processor produces less accurate output. Since PR #350 introduced configurable processor selection, we will change the behaviour to expose a list of available processors for each document type instead of separating them into "standard" and "fast" versions. |
| output_path=output_path, | ||
| use_fast_processors=False, | ||
| # use pymupdf4llm by the api by default, should be configurable in the future | ||
| use_fast_processors=True, |
There was a problem hiding this comment.
Fast processing will apply to other file types aside from pdf. Is it the intent?
There was a problem hiding this comment.
Yes, this is a temporary "hack" that was discussed with @fabnemEPFL. This is because for the moment, the separation between processor is done using normal mode or fast_mode, so enabling fast_mode is the only way to use pymupdf4llm in the index api for the moment. We should later fix the current processor selection to create a list of available processors instead of separating them by the normal or fast mode.
…tical module order


pymupdf has an extension named pymupdf4llm that can convert documents into structured Markdown, JSON, and plain text in a suitable way for RAG pipelines : https://github.com/pymupdf/pymupdf4llm.
There is now a new pdf processor
src/mmore/process/processors/pdf_pymupdf4llm_processor.pythat extract pdf in a markdown structure in fast mode.It can be configured by setting :
in a process config file.