Repository navigation
fix(crawl): report the real timeout and stop crawling binary assets - #619
Merged
Merged
Conversation
- a render budget that expired with nothing extractable reported the time of the partial render as the timeout (`Timeout after 2579ms` on a 30s request); it now reports the caller's budget, like every other timeout path - crawl skips same-origin links to images, media, archives, fonts and installers instead of fetching each body only to return an unsupported content placeholder and spending a slot of the caller's `limit` on it; map still lists those URLs but no longer fetches them for links - map discovery waits for in-flight pages no longer than its own deadline, so a page that overruns its budget no longer holds the BFS open until the route timeout drops every URL already found (504 with zero links)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes from production logs (2026-09-29 to 2026-10-13).
Timeout message reports the caller's budget
scrape_url_innerreturnedTimeout(fetch_result.elapsed_ms), the moment the partial render was taken. A request withtimeout: 30000answered after ~33s saidTimeout after 2579ms; on the Firecrawl surface that text reaches the caller (SCRAPE_TIMEOUT). 292 timeouts in the window carried a 2.5 to 5s figure.Timeout(deadline.requested_ms()), as every other timeout path already does. The WARN line keepselapsed_ms. Status codes are unchanged (v1 504, v2 408).Crawl skips binary assets
a[href], and gallery pages link straight to.jpg. Crawls fetched 638 image/video/archive links (250 MB in 34h), each read in full only to return an unsupported-content placeholder, and each used up a slot of the caller'slimit.is_static_asset(last path segment's extension: images, media, archives, fonts, installers) skips them in crawl enqueue.pdfand office documents, which are parsed, are not in the list; neither arejs/css, so slugs like/topics/node.jsstill crawl. Map still lists asset URLs but no longer fetches them for links.Map discovery returns at its deadline
/v2/mapresponses were 504 with zero links at exactly 45s.timeout_at(bfs_deadline_at, ...), so discovery returns what it found. Newtests/discover_deadline.rsstalls a page body past the deadline: the old loop overshot by ~457ms, the new one by ~4ms.No API shape change, no config change. Crawl results stop containing placeholder entries for asset links.