chore(release): promote pre/beta to main (v2.2.4-beta.1) - #1146
Merged
Conversation
## [2.2.0-beta.7](v2.2.0-beta.6...v2.2.0-beta.7) (2026-08-21) ### Bug Fixes * resolve markdown links from the document URL ([875385b](875385b)) ### Docs * fix timeout documentation links ([4763fdc](4763fdc))
…ing NA A page that could not be scraped as intended was indistinguishable from one that could. FetchNode's default path (ChromiumLoader -> ascrape_playwright) dropped the Response returned by page.goto(), so a 404, 403, 500, captcha wall or login redirect reached the LLM as ordinary content and the model answered "NA" with nothing in the logs to explain why. Reported in #1102, where en.wikipedia.org/wiki/Timpson_(company) 404s (the article is at Timpson_(retailer)) and the run still looked clean. Two deterministic, LLM-free guards, both warnings so existing behaviour is unchanged for anyone deliberately scraping error pages: - ChromiumLoader keeps the Response from every page.goto() call site (ascrape_playwright, ascrape_playwright_scroll, ascrape_with_js_support) and warns on status >= 400. This mirrors what the opt-in use_soup=True path in FetchNode has always done. - ParseNode warns when the parsed content contains none of the terms the user asked about — schema field names plus the significant words of the prompt. A 200 response can still reach the LLM without the requested data: content behind JavaScript that never rendered, a field inside a <script> blob the parser drops, or a document truncated beyond the model window. Zero matches is a deliberately conservative bar, so the warning stays quiet when the page simply phrases the answer differently. The graphs now pass their schema to ParseNode so it has the field names available. Verified against the URLs from the issue: the 404 now logs "Received HTTP 404 for .../Timpson_(company); the scraped content is likely an error page" before returning NA, while the corrected URL stays silent and answers 1865. Also drops three dead imports from smart_scraper_multi_batch_graph.py, which ruff blocks on now that the file is touched. Fixes #1102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e-detection fix(fetch): surface HTTP errors and missing content instead of answering NA
… the config file
`_check_config_and_environ_for_telemetry_flag` checked that the environment
variable existed and then read its value out of the config file:
if os.environ.get("SCRAPEGRAPHAI_TELEMETRY_ENABLED") is not None:
try:
telemetry_enabled = config_obj.getboolean("DEFAULT", "telemetry_enabled")
except Exception:
pass
With no `telemetry_enabled` key in `~/.scrapegraphai.conf`, `getboolean` raises,
the bare `except` swallows it, and the flag keeps its default of `True`. So the
opt-out documented in the README leaves telemetry on for anyone who has not also
written a config file.
Now parses the variable's own value, reusing `configparser`'s BOOLEAN_STATES so the
environment variable and the config file accept the same spellings (true/false,
yes/no, on/off, 1/0). An unparseable value logs a warning and leaves the flag alone
rather than failing silently.
Adds tests/test_telemetry_flag.py covering the config path, the environment path,
precedence between them, and an unparseable value.
Verified with SCRAPEGRAPHAI_TELEMETRY_ENABLED=false and no config key:
before: True after: False
…ni-2.5-tokens fix(models): add Gemini 2.5 token limits
## [2.2.0-beta.9](v2.2.0-beta.8...v2.2.0-beta.9) (2026-09-07) ### Bug Fixes * **models:** add Gemini 2.5 token limits so they are not truncated to 8192 ([c21af20](c21af20))
fix: 🐛 read SCRAPEGRAPHAI_TELEMETRY_ENABLED from the environment, not the config file
## [2.2.0-beta.10](v2.2.0-beta.9...v2.2.0-beta.10) (2026-09-07) ### Bug Fixes * 🐛 read SCRAPEGRAPHAI_TELEMETRY_ENABLED from the environment, not the config file ([8769c3b](8769c3b))
Brings v2.2.1-v2.2.3 from main into pre/beta, including the proxy rotation fix (#1145). Conflicts resolved by keeping pre/beta's pre-release version in pyproject.toml and main's stable CHANGELOG, both of which semantic-release regenerates on the next release.
## [2.2.4-beta.1](v2.2.3...v2.2.4-beta.1) (2026-09-07) ### Bug Fixes * 🐛 read SCRAPEGRAPHAI_TELEMETRY_ENABLED from the environment, not the config file ([8769c3b](8769c3b)) * **models:** add Gemini 2.5 token limits so they are not truncated to 8192 ([c21af20](c21af20)) * **fetch:** surface HTTP errors and missing content instead of answering NA ([f91478e](f91478e)), closes [#1102](#1102) [#1102](#1102) ### CI * **release:** 2.2.0-beta.10 [skip ci] ([0bb8bc9](0bb8bc9)) * **release:** 2.2.0-beta.7 [skip ci] ([decfc6b](decfc6b)) * **release:** 2.2.0-beta.8 [skip ci] ([d59c3df](d59c3df)), closes [#1102](#1102) [#1102](#1102) * **release:** 2.2.0-beta.9 [skip ci] ([3047ef8](3047ef8))
|
🎉 This PR is included in version 2.2.4 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
pre/betatomain.pre/betais now strictly ahead ofmain(0 behind), so this merges without conflicts.What this brings to main
SCRAPEGRAPHAI_TELEMETRY_ENABLEDfrom the environment, not the config file (fix: 🐛 read SCRAPEGRAPHAI_TELEMETRY_ENABLED from the environment, not the config file #1141) — the documented opt-out was a no-op on a fresh install: the check tested that the env var existed but then read the value out of~/.scrapegraphai.conf, so with notelemetry_enabledkeygetbooleanraised, the bareexceptswallowed it, and the flag stayedTrue. The value is now parsed from the variable itself viaConfigParser.BOOLEAN_STATES, so env and config accept the same spellings. Addstests/test_telemetry_flag.py(20 cases).main.Merge branch 'main' into pre/beta— brings v2.2.1–v2.2.3 (including the proxy rotation fix, fix(proxy): handle schemeless proxy server format and broker routing #1145) down onto the beta line. Conflicts were onpyproject.toml(version) andCHANGELOG.mdonly, resolved by keeping the pre-release version and main's stable changelog; semantic-release regenerated both and cut v2.2.4-beta.1.Verification
Both new/affected suites pass locally —
tests/test_telemetry_flag.py(20 passed) andtests/utils/test_proxy_rotation.py(9 passed). Note that neither file is in thetest-suite.ymlrun list, so CI does not cover them:test_proxy_rotation.pystill makes live calls to free-proxy-list.net intest_search_proxy_servers_success/_exception, buttest_telemetry_flag.pyis fully deterministic and is a good candidate to add to the CI list in a follow-up.Release workflow and Test Suite are green on
pre/betaat this head.