fix(scraper): keep crawling while one page is slow

[arabold]

Fixes #490, which was reopened after 3.2.0.

What the reporter sees

After 3.2.0 the crawl of https://www.brendangregg.com still seems to stop, at Guess/guess.qbas when **/blog/** is excluded and at Guess/guess.ps when it is not.

I reproduced this on 3.2.1 with both the CLI and the web UI (with the UI's default exclude patterns). The crawl never actually hangs. It always finishes (621/621 processed, 440 indexed), but it freezes for about 7 seconds right after guess.qbas.

Why it freezes

Three files in that directory always answer HTTP 500, because Apache tries to run them as CGI scripts: guess.py, guess.pl and guess.pl.txt. The fetcher treats 500 as retryable, so each one takes 4 attempts with 1 + 2 + 4 seconds of backoff in between.

The crawl loop processed pages in fixed batches of maxConcurrency (3) and waited for the whole batch before starting the next one. So while one URL was waiting to retry, the other two slots did nothing and the whole crawl stood still.

The Jobs page shows the last finished URL as "Processing …". In the directory listing, guess.py sits between guess.ps and guess.qbas, so the page that looks stuck depends on how the batch of 3 lines up. Excluding the blog or not changes that alignment, which matches the report exactly. The same batch barrier was also behind part of the original "freezes for a long time" report, when a large image download held up its batch.

The change

The loop in BaseScraperStrategy.scrape() now runs a sliding window. When an item finishes, the next queued item starts right away. A URL waiting to retry now only holds up its own slot.

Two things stay the same on purpose:

• Crawl order. Links found by a page are added to the queue in the order pages were taken off it, not in the order they finish. The queue stays breadth-first, so every URL is still first offered at its shortest depth, and timing does not change which pages are crawled or which pages fit under maxPages.
• The maxPages limit. Items that are still running count against the remaining budget, so at most maxPages - pagesIndexed items run at once and the limit is never overshot. This is the same guarantee the batch size gave before.

processBatch is split into processQueueItem (one item to its outcome) and admitToQueue (deduplication and the queue totals). Most of the diff in BaseScraperStrategy.ts is the re-indentation of that body, so reviewing with whitespace hidden is easier. The two WebScraperStrategy tests that called processBatch directly now call processQueueItem.

Tests

Four new tests in BaseScraperStrategy.test.ts:

• A slow page no longer blocks the pages queued behind it. This test fails on main.
• Each URL keeps its shortest depth when pages finish out of order. main already passes this. It guards the admission order: I checked that it fails if links are added in finish order, which the existing breadth-first test does not catch.
• A slot freed by a page that stored nothing is refilled without indexing past maxPages.
• The crawl does not finish while a page that reached maxPages is still being stored. This was found in review: the first version of the loop stopped as soon as the count reached the limit, so scrape() could resolve with a write still pending. The loop now runs until nothing is left running, and reaching maxPages only stops new items from starting.

Lint and a cold-cache typecheck are clean, and all 2329 tests pass locally.

Live check

I ran the same web UI crawl of brendangregg.com before and after the change:

	before	after
pages processed	621 / 621	621 / 621
pages indexed	440	440 (identical URL set)
gap after guess.qbas	7 s	none, all of Guess/ stores within one second
total time	159 s	143 s

After the change, the longest time the Jobs page went without updating was 3.1 seconds, on a slide page that has to be rendered in a browser.

Not changed here

• The Jobs page label "Processing …" names a page that has already finished. That is what made an innocent page look stuck. It is a small separate UI change.
• Whether a 500 should be retried three times is a policy question. Retries now cost only one slot, so I left the policy as it is.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

The crawl can resolve before all in-flight progress callbacks and storage operations finish.

Review effort: Balanced
Findings: 1

Open (1)

• Wait for in-flight progress callbacks before completing scrape · New

What changed in this PR

Replaces fixed-batch scraping with a sliding concurrency window while preserving crawl order and page limits.

Changes:

• Splits item processing from ordered queue admission.
• Adds concurrency, breadth-first ordering, and limit tests.
• Updates direct strategy tests for the new method.

File	Description
BaseScraperStrategy.ts	Implements sliding-window crawling.
BaseScraperStrategy.test.ts	Adds concurrency regression coverage.
WebScraperStrategy.test.ts	Updates internal method calls.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.