Skip to content

feat(renderer): fresh impersonated session per hop attempt - #613

Draft
paoloantinori wants to merge 2 commits into
fastcrw:mainfrom
paoloantinori:impersonated-fresh-session-per-hop
Draft

paoloantinori wants to merge 2 commits into
fastcrw:mainfrom
paoloantinori:impersonated-fresh-session-per-hop

Conversation

@paoloantinori

Copy link
Copy Markdown
Contributor

What

Each impersonated hop attempt now runs on a fresh, use-once client: new TLS session, no pooled identity. The pin arm keeps the warm pooled client.

Why

A pattern arrived from outside the project. Waze blocked the unofficial endpoint behind Home Assistant's Waze Travel Time on 2026-09-30, and pywaze's recovery documents a lesson this tier should absorb:

crw's hop currently fetches through one shared wreq::Client built at startup (plus warm per-proxy clients). If a wall vendor tracks per-fingerprint reputation, a busy crw instance retrying the same walled host through that shared client accumulates reputation, and the tier stops winning for that host without any error.

Honest caveat: this is insurance, not an observed bug here. The walls measured for this tier (the Amazon.it class) behave per-request intermittent and IP/rate-based; no fingerprint-reputation wall is in the dataset. Two supporting observations from the live runs while preparing this change: the wall has been seen to hit the impersonated client on isolated single requests (the direct live test failed once on a wall-shaped body and passed on retry, in the same run where the chain e2e passed), and preset aging is already handled by graceful fall-through plus the sentinel live tests. Still, the hardening costs nothing where it applies: the hop is by construction a cold retry path, so a per-attempt client costs one TLS handshake and no pooling benefit is lost.

The split matches the two documented roles of the module. The hop is a reactive retry, so freshness wins; the pin is deliberate bulk use, so pooling wins.

Shape

  • New ImpersonatedHopFactory type alias and impersonated_hop_factory field on FallbackRenderer: production builds a fresh ImpersonatedFetcher per call, honoring REQUEST_PROXY fail-closed through the same with_proxy constructor the warm path uses.
  • try_impersonated_hop takes its fetcher from the factory instead of the shared lookup; the pin arm still uses impersonated_fetcher_for_request (doc updated to say pin-only).
  • The factory is a field rather than a method so tests can inject a counter: hop_builds_a_fresh_session_per_attempt asserts exactly one construction per hop attempt (wall miss falls through to the ladder), clean_site_builds_no_hop_session asserts zero on clean sites. The eleven existing hop tests inject through a small inject_impersonated helper that sets both the presence field and the factory.

Verification

cargo test -p crw-renderer --features impersonated (796 passed, 4 ignored live), lean cargo check -p crw-renderer (no wreq in the tree), cargo fmt --check, and cargo clippy -p crw-renderer --features impersonated,camoufox,cloak,cdp -- -D warnings all run locally. The four live #[ignore] tests run against the real Amazon wall (one single-attempt flake of the direct tier test on a walled request, passing on retry; the auto-chain e2e green in the same run).

Marked draft because the design question deserves maintainer eyes: is per-fingerprint reputation a threat class worth hardening against now (this change), or should the tier stay on the pooled client until a wall is observed doing it (in which case closing this is also fine)?

The hop is a cold retry path, so it now builds a use-once
ImpersonatedFetcher per attempt: new TLS session, no pooled identity. A
persistent impersonated session is exactly what lets a wall vendor build
per-fingerprint reputation and quietly stop serving it, the failure pywaze
paid to learn (eifinger/pywaze#110: a pinned impersonated session was
re-blocked in six days; fastcrw#111 fixed it with a fresh session per fallback
request). The pin arm keeps the warm pooled client, where pooling pays.
REQUEST_PROXY stays honored fail-closed through the same with_proxy
constructor.

The builder is a factory field (type ImpersonatedHopFactory) so tests can
inject a counter: hop_builds_a_fresh_session_per_attempt asserts exactly
one construction per hop attempt, clean_site_builds_no_hop_session asserts
none on clean sites.

Gates: /simplify, code-review
Asserts the pinned Chrome149 preset carries the post-quantum key share
(X25519MLKEM768) in both supported_groups and the actual key_share, plus
an HTTP/2 fingerprint hash, by probing a fingerprint-echo endpoint
(tls.peet.ws/api/all). A preset that claims Chrome 149 without the PQ
share is detectable by post-quantum-aware walls before any HTTP traffic;
when this test goes red the answer is a preset bump, not a code fix.
Deterministic: reads the handshake shape, not a wall that may not fire.

Gates: /simplify, code-review
@paoloantinori

Copy link
Copy Markdown
Contributor Author

Design note from follow-up research (claudedocs/research_impersonation_landscape_2026-10-11.md in the fork, not linked): the fresh-session-per-hop choice is correct for this tier's population, and it would be wrong for bulk. Recording the boundary here so it is not silently widened later.

Anti-bot literature in 2026 treats identity churn from one IP as a bot signal in itself. Incogniton documents that a never-seen fingerprint is suspicious. Pre-HTML detection evaluates browser consistency above the handshake. Translated: rotating the impersonated identity per-request at high volume from one address tells the wall you are not a person. pywaze went fresh-per-request because Waze's wall punishes continuity, but that is the minority case.

The split this PR implements is the safe subset: the hop is a cold retry that fires once per walled page, so fresh-per-attempt costs nothing and cannot create churn. The pin arm keeps the pooled client because deliberate bulk use is where identity consistency wins. If the pin ever grows per-host bulk crawling, the correct next step is a sticky per-host identity (client + cookie jar + slow TTL rotation), not widening fresh-per-attempt to the pin.

Also pushed as a second commit: a deterministic sentinel for the preset's ClientHello shape (bd15a26), asserting the post-quantum key share and HTTP/2 fingerprint from a fingerprint-echo endpoint. This catches preset aging (a Chrome preset that stops carrying X25519MLKEM768 is detectable by post-quantum-aware walls) without depending on a wall firing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant