Context
During initial deployment (docs/DEPLOYMENT_DAYBREAK_SERVER.md),
6 sources were found failing on every fetch. One was a genuine, fixed
bug (Sophos News' feed URL had moved — corrected to
https://www.sophos.com/en-us/blog/feed, confirmed working).
The remaining 4 are external issues, not fixable with confidence from
our side:
- CISA Advisories, Cybernews:
403 at the whole-domain level (even
the homepage 403s with a real browser User-Agent from this box's IP)
— WAF/bot-detection blocking, likely tied to the Linode datacenter
ASN. CISA content isn't fully lost — CISA KEV is a separate,
healthy adapter using a different, non-WAF-fronted API.
- The Record, The Register: succeeded in isolated manual
curl
tests, still failed on two separate forced re-fetches through the
app's actual FeedFetcher minutes apart. Intermittent bot-challenge
scoring, not reliably reproducible.
- Datenschutz-Guru: consistent
500 from their own nginx — broken
on their end, not us. (Not re-investigated as deeply since it's
clearly not our problem; included here for completeness.)
An inconclusive lead, worth revisiting carefully
FeedFetcher::get() forces CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_2 (TLS 1.2 only, no 1.3). In one isolated test,
forcing TLS 1.2 got CISA Advisories through (200) while the identical
setting failed via the app moments earlier — a signature of
fingerprint-based bot scoring that fluctuates request-to-request, not
a deterministic bug. Did not change this — it's used by all 25+
sources, and one contradictory data point isn't enough confidence to
risk breaking currently-working sources to chase a flaky one.
Task
Revisit periodically (or if someone wants to invest more time):
monitor whether these sources self-recover via the app's normal
degraded-status retry cadence, and/or carefully A/B test the TLS
version behavior in isolation (e.g. a feature flag or per-source
override) before ever changing it globally.
Context
During initial deployment (docs/DEPLOYMENT_DAYBREAK_SERVER.md),
6 sources were found failing on every fetch. One was a genuine, fixed
bug (Sophos News' feed URL had moved — corrected to
https://www.sophos.com/en-us/blog/feed, confirmed working).The remaining 4 are external issues, not fixable with confidence from
our side:
403at the whole-domain level (eventhe homepage 403s with a real browser User-Agent from this box's IP)
— WAF/bot-detection blocking, likely tied to the Linode datacenter
ASN. CISA content isn't fully lost —
CISA KEVis a separate,healthy adapter using a different, non-WAF-fronted API.
curltests, still failed on two separate forced re-fetches through the
app's actual
FeedFetcherminutes apart. Intermittent bot-challengescoring, not reliably reproducible.
500from their own nginx — brokenon their end, not us. (Not re-investigated as deeply since it's
clearly not our problem; included here for completeness.)
An inconclusive lead, worth revisiting carefully
FeedFetcher::get()forcesCURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_2(TLS 1.2 only, no 1.3). In one isolated test,forcing TLS 1.2 got CISA Advisories through (200) while the identical
setting failed via the app moments earlier — a signature of
fingerprint-based bot scoring that fluctuates request-to-request, not
a deterministic bug. Did not change this — it's used by all 25+
sources, and one contradictory data point isn't enough confidence to
risk breaking currently-working sources to chase a flaky one.
Task
Revisit periodically (or if someone wants to invest more time):
monitor whether these sources self-recover via the app's normal
degraded-status retry cadence, and/or carefully A/B test the TLSversion behavior in isolation (e.g. a feature flag or per-source
override) before ever changing it globally.