Harden remote-file ingestion and inbound endpoints#586
Merged
Conversation
Closes #573. Remote downloads (Meedan images, Airtable documents): - Require HTTPS and reject private, loopback, link-local, CGNAT, cloud-metadata, and other non-public addresses for the initial URL, every DNS result, and every redirect destination (src/utils/ssrf.ts) - Follow redirects manually with a bounded hop count, add response and total-download timeouts, and cap concurrent downloads per instance - Meedan image URLs must match the MEEDAN_ALLOWED_IMAGE_HOSTS allowlist (defaults to checkmedia.org,*.checkmedia.org) Uploads (/api/airtable-upload): - Extension and MIME metadata must both be allowed and consistent, and file magic bytes must match the claimed type (src/utils/fileSignature.ts) - SVG is no longer accepted from any ingestion path; locally served media gets nosniff + sandboxing CSP headers - Per-IP rate limit, concurrency gate, and rolling daily byte budget Inbound endpoints: - Meedan webhook secret compared in constant time - Duplicate webhook deliveries are idempotent: completed responses replay from a body-fingerprint cache, in-flight duplicates get 409, and 5xx outcomes stay retryable - Meedan webhook rate limiting per client IP Quotas and env vars are documented in docs/remote-file-ingestion.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
… sandbox CSP The global headers rule from #585 applies first; the /api/media/file/* rule follows so its sandboxing CSP overrides the site-wide policy for served upload files (nosniff already comes from the global rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #573
What this does
Creates a safe boundary for Meedan remote downloads and Airtable uploads, and adds abuse controls to state-changing inbound endpoints.
Remote downloads (
src/utils/files.ts,src/utils/ssrf.ts)MEEDAN_ALLOWED_IMAGE_HOSTSallowlist (defaultcheckmedia.org,*.checkmedia.org), applied in both the webhook route and the workflow sync.169.254.169.254metadata), CGNAT, multicast, reserved, and IPv6-equivalent (mapped/NAT64/6to4/ULA) addresses are rejected for IP-literal hosts, for every DNS result, and for every redirect destination (redirects are followed manually and re-validated per hop).DOWNLOAD_MAX_REDIRECTS, default 3), response-header timeout per hop (30s), total download deadline (5min), streaming byte limit, and a per-instance download concurrency semaphore (default 4).Uploads (
/api/airtable-upload)src/utils/fileSignature.ts) — spoofed files are rejected with 415./api/media/file/*) now getsX-Content-Type-Options: nosniffand a sandboxing CSP.Inbound endpoints
safeCompare, also used by the upload token check).Docs
Quotas, env vars, deployment notes (per-instance guard semantics), and residual-risk notes are documented in
docs/remote-file-ingestion.md;.env.exampleupdated.Behaviour changes to be aware of
extractDocumentskeeps fetching the app's own media fromNEXT_PUBLIC_APP_URLvia an explicit internal-only exception.Testing
Share.tsxerrors untouched).🤖 Generated with Claude Code