Route ScrapeGraphAI LLM scraping graphs through ProxyHat residential proxies — a geo-targeted, sticky-by-default residential IP dropped straight into your graph_config.
Tip
Recommended proxies — ProxyHat residential IPs. Every feature in this package is tested end-to-end against ProxyHat and works great. First-class integration; also works with any proxy, or none.
ScrapeGraphAI drives a real browser (Playwright) to render pages before the LLM reads them. From a datacenter IP that browser gets flagged, CAPTCHA-walled, and blocked. This package plugs ProxyHat's residential IPs (50M+ across 148+ countries) into ScrapeGraphAI through the exact hook it already supports — graph_config["loader_kwargs"]["proxy"] — with one pinned residential IP for the whole scrape by default, so pagination and JS interactions stay on one consistent identity. No fork, no boilerplate.
pip install scrapegraphai-proxyhat scrapegraphaiscrapegraphai is a heavy peer dependency (LLM + Playwright stack) — bring your own version. The config builders (proxyhat_graph_config(), proxyhat_proxy()) work on their own; you only need scrapegraphai installed to actually run a graph.
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai_proxyhat import proxyhat_graph_config
# An API key auto-selects an active residential sub-user:
graph_config = proxyhat_graph_config(
{"llm": {"model": "openai/gpt-4o-mini"}},
country="us", # sticky US residential IP for the whole scrape
)
scraper = SmartScraperGraph(
prompt="List every project with its title and description.",
source="https://perinim.github.io/projects/",
config=graph_config,
)
print(scraper.run())Get an API key at proxyhat.com.
proxyhat_graph_config(base_config, ...) merges the proxy into loader_kwargs.proxy and preserves everything else in your config (llm, verbose, headless, any existing loader_kwargs). Your base_config is never mutated.
Just want the proxy dict for a config you build yourself?
from scrapegraphai_proxyhat import proxyhat_proxy
graph_config = {
"llm": {"model": "openai/gpt-4o-mini"},
"loader_kwargs": {"proxy": proxyhat_proxy(country="de", sticky="1h")},
}
# proxyhat_proxy(...) -> {"server": "http://gate.proxyhat.com:8080", "username": ..., "password": ...}Or the raw URL string (proxyhat_proxy_url(...)) for any other tool:
from scrapegraphai_proxyhat import proxyhat_proxy_url
url = proxyhat_proxy_url(country="us")
# http://<user>-country-us-sid-<id>-ttl-30m:<pass>@gate.proxyhat.com:8080Pass them explicitly or via environment variables — options win over env:
| Option | Env var | Notes |
|---|---|---|
api_key |
PROXYHAT_API_KEY |
Auto-selects an active sub-user with remaining traffic |
sub_user |
PROXYHAT_SUBUSER |
Pick a specific sub-user by uuid or name (with an API key) |
username |
PROXYHAT_USERNAME |
Explicit gateway proxy_username (skips the API) |
password |
PROXYHAT_PASSWORD |
Explicit gateway proxy_password |
proxyhat_graph_config(
base_config,
protocol="http", # or "socks5"
country="us", # ISO code or "any" (default)
region="california",
city="los_angeles",
filter="high", # AI IP-quality tier
sticky="30m", # session lifetime (default); sticky=False rotates every request
)All three builders (proxyhat_graph_config, proxyhat_proxy, proxyhat_proxy_url) take the same targeting keyword arguments.
A single scrape often hits the same site many times — rendering, scrolling, following pagination, retrying. If the exit IP changed mid-scrape the site would see a visitor teleporting between cities and block it. So this package is sticky by default: one residential IP is pinned for the whole run (sticky="30m"), keeping the session coherent.
Want a fresh IP on every request instead (e.g. a SearchGraph fanning out across many independent pages)? Turn stickiness off:
proxyhat_graph_config(base_config, country="us", sticky=False) # rotating residential IP per connectionSet a custom lifetime with sticky="2h".
ScrapeGraphAI reads a proxy at graph_config["loader_kwargs"]["proxy"] and hands it to its ChromiumLoader (and on to Playwright) as a Playwright-style {server, username, password} mapping. proxyhat_proxy(...) resolves your gateway credentials (via the official proxyhat SDK — an API key auto-picks an active sub-user, or pass username/password), then builds exactly that mapping: server is the ProxyHat gateway (http://gate.proxyhat.com:8080, or socks5://…:1080) and username carries ProxyHat's targeting grammar (<user>-country-us-sid-<id>-ttl-30m). proxyhat_graph_config(...) just merges that dict into loader_kwargs.proxy over your config. A sticky username pins one residential IP; a rotating one makes the gateway hand out a fresh IP per connection.
MIT © ProxyHat