Skip to content

fix(crawl): report the real timeout and stop crawling binary assets - #619

Merged
us merged 1 commit into
mainfrom
fix/timeout-message-and-crawl-assets
Oct 13, 2026
Merged

us merged 1 commit into
mainfrom
fix/timeout-message-and-crawl-assets

Conversation

@us

@us us commented Oct 13, 2026

Copy link
Copy Markdown
Member

Three fixes from production logs (2026-09-29 to 2026-10-13).

Timeout message reports the caller's budget

  • When a render budget expired with nothing extractable, scrape_url_inner returned Timeout(fetch_result.elapsed_ms), the moment the partial render was taken. A request with timeout: 30000 answered after ~33s said Timeout after 2579ms; on the Firecrawl surface that text reaches the caller (SCRAPE_TIMEOUT). 292 timeouts in the window carried a 2.5 to 5s figure.
  • Now Timeout(deadline.requested_ms()), as every other timeout path already does. The WARN line keeps elapsed_ms. Status codes are unchanged (v1 504, v2 408).

Crawl skips binary assets

  • Link discovery takes every same-origin a[href], and gallery pages link straight to .jpg. Crawls fetched 638 image/video/archive links (250 MB in 34h), each read in full only to return an unsupported-content placeholder, and each used up a slot of the caller's limit.
  • is_static_asset (last path segment's extension: images, media, archives, fonts, installers) skips them in crawl enqueue. pdf and office documents, which are parsed, are not in the list; neither are js/css, so slugs like /topics/node.js still crawl. Map still lists asset URLs but no longer fetches them for links.

Map discovery returns at its deadline

  • The BFS awaited in-flight pages with no deadline of its own and only checked the clock after a page landed. A page whose recovery tier overran its budget held the loop open until the route's outer timeout fired and dropped every URL already found: 32 /v2/map responses were 504 with zero links at exactly 45s.
  • The wait is now timeout_at(bfs_deadline_at, ...), so discovery returns what it found. New tests/discover_deadline.rs stalls a page body past the deadline: the old loop overshot by ~457ms, the new one by ~4ms.

No API shape change, no config change. Crawl results stop containing placeholder entries for asset links.

- a render budget that expired with nothing extractable reported the time of
  the partial render as the timeout (`Timeout after 2579ms` on a 30s request);
  it now reports the caller's budget, like every other timeout path
- crawl skips same-origin links to images, media, archives, fonts and
  installers instead of fetching each body only to return an unsupported
  content placeholder and spending a slot of the caller's `limit` on it;
  map still lists those URLs but no longer fetches them for links
- map discovery waits for in-flight pages no longer than its own deadline, so
  a page that overruns its budget no longer holds the BFS open until the
  route timeout drops every URL already found (504 with zero links)
@us
us merged commit df29f00 into main Oct 13, 2026
10 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant