Known collection and processing issues

Use this register to understand collection-quality problems, the handling already in place, and the gaps that remain. Detailed commands live in the linked feature guides; live outcomes live in run history and logs. Track outstanding work and future ideas in Linear.

Open-work consolidation: 2026-10-27. Observations below retain their original dates. Publisher examples describe observed captures, not a claim that a site always fails. “Mitigated” means a fallback exists; “handled” means the specific case has a tested fix.

Browser protocol compatibility

The 2026-11-04 log contains 741 localNetworkAccessRequestPolicy parsing errors: Chromium 138 sends privateNetworkRequestPolicy, while Zendriver 0.17.1 expects the newer Local Network Access schema. Zendriver drops these Network.requestWillBeSentExtraInfo events before dispatching handlers. The 302 historical sameParty errors last appeared on 2026-10-01; the pinned Zendriver version already accepts cookies without that removed field.

Before starting a browser, DANK installs a narrow parser compatibility fix. Legacy request events retain their cookies, headers, timing and request ID; their optional client security state is omitted because the old preflight policies have no exact equivalent in the newer permission-based schema. DANK does not use that metadata. Modern security states are preserved, and unrelated schema errors still surface. This changes event decoding only, without changing browser security settings or site access restrictions. See protocol regression tests for both schemas, cookies with and without sameParty, and repeated installation. Replaying all 1,043 logged protocol-error events through the installed driver reproduced 741 errors before the fix and none after it, without browser traffic. Other browser/driver schema differences can still require separate fixes.

Reader refresh results

Refresh results start collapsed below the header and expand within the page, without covering reading controls. The summary flags partial, failed and interrupted results as needing attention. A result with saved content can also be ready to view; this does not mean collection succeeded completely. Submission, progress and update-loading errors stay visible while details are collapsed. See refresh browser tests.

Source edits in config.toml require an app restart

With sources_in_database off, the default, the reader loads sources from config.toml at startup and passes that snapshot to refresh jobs. Docker also copies the mounted TOML into the container at startup. Editing the host file does not update the sidebar or a refresh launched by the running app. A successful refresh therefore does not mean newly added sources were attempted.

On 2026-10-26, X run ea85d21b-e2e9-4369-b75e-febdc7335b82 saved 1,343 posts from the previous 37-account list after JDM4NC was added to the host config. The new account had no run entry or saved posts. DANK's internal source resolver accepted it from the host config but rejected it from the container's old copy. Recreating the app and refreshing only that account saved 28 posts, including 21 owned by the account; the reader returned 20 in its default Posts view.

After source edits, recreate Docker's app with make web ARGS='--force-recreate dank', or restart the native viewer, then refresh the new source. Or move the sources into Postgres: with sources_in_database on, each request, refresh and command reads them as it starts, so saved edits apply without a restart. Refreshes already queued or running keep the sources they were submitted with, and an outage is an error rather than a fallback to config.toml. Other reader tabs pick up edits within five seconds. See saved source tests and refresh job tests.

Blocked Reddit collection and cancellation

The 2026-10-19 run c01507f1 saved 3,044 posts, but spent its last 33 minutes waiting on repeated Reddit network-security blocks after the other 56 sources finished. Five-minute cooldowns and a fresh retry for each community compounded the delay. Ctrl+C reached cleanup, but browser-driver shutdown did not finish.

An explicit Reddit network-security block or HTTP 429 now defers further Reddit requests for the current run, including later Reddit source entries. Saved posts are retained and queued media can finish; other sources and final enrichment continue. The source remains partial or failed. Other shared cooldowns have a 30-second wait ceiling. An unusable initial listing still gets one ten-second retry; ordinary community restrictions do not stop unrelated communities. This bounds the failure without bypassing Reddit's access restrictions.

Browser shutdown bounds the graceful close, transport close and driver cleanup to five seconds each, with a final kill of the owned process if necessary. Cancellation never enters the native browser's keep-open debugging mode. Final run history is written after cleanup; already interrupted historical runs can still show running when their final row was never saved. See Reddit recovery tests and browser shutdown tests.

Controlled Docker verification on 2026-10-19 used real Chromium and an isolated ClickHouse database with simulated sources. The blocked run finished partial in 1.0 seconds with both saved posts retained and the other source completed. Real Ctrl+C removed both normal and deliberately stalled-browser containers; both runs recorded cancelled, preserving their two saved posts.

Linked collection and discovery timing

Optional Reddit publisher collection is controlled by [reddit].collect_linked_articles (default false). It reuses readable saved articles, saves Reddit discoveries before publisher requests and retains explicit origin metadata for new fetches. Backfill is bounded by configured community post limits. A failed or unsupported publisher page remains unavailable.

The Info icon opens collection details regardless of earlier references. A separate eye requires independent discovery before the post's verified publication time. Its tooltip includes the gap from the displayed discovery to publication, using the graph’s duration format. Earlier collection order alone does not qualify. Current-post origins, tied timestamps and unknown publication/collector data do not produce that signal. Native video playback alone also does not produce an eye.

Collection info reports first observed article/link times independently of successful page collection, and does not infer missing legacy collector routes. Processed-version cutoffs prevent a pending raw capture from upgrading an older displayed article. Edited posts compare the current link's first observation, not the post identity's original timestamp. Polling and selection bias remain; these observations do not establish publication priority or copying.

Connections scopes referencing posts to the active source/community; All and Blogs support broader exploration. Article-first layout is conceptual, not a chronological or causal claim. The graph's First discovered annotation shows saved source/time history without expanding that source's other references; its dashed connection is distinct from hyperlink arrows and reference counts. The current social post shows verified publication and its gap from discovery, using neither post collection nor article download time as a publication fallback. Before/after, exact ties and unavailable timing remain explicit. See feature details, the source novelty proposal, collection tests, and provenance tests.

Blog source icons

The reader captures a source favicon when its sidebar icon first loads, without collecting articles. It checks the source homepage and a bounded set of declared raster icons, then /favicon.ico; copies are resized and saved locally as PNG. Unknown sources and private network destinations are rejected. Up to three captures run at once, with a 15-second budget per source and image size limits.

Successful icons are reused for 30 days. Failures retry after six hours while the web app remains running, and a failed refresh preserves an older good icon. Sources offering only SVG icons, blocked pages or unsupported images keep the reader fallback. See source icon tests.

Publisher excerpts across feeds

Reddit link posts with only a URL and preview image now use a matching saved publisher excerpt. The same fallback applies to empty RSS aggregator entries and missing descriptions on link-only X previews. HN keeps its existing fallback; YouTube video cards have no publisher-article destination to borrow from.

Matches use exact URL identities and observed redirects, with one batched lookup. Own post text, captions and captured X descriptions take precedence. Multiple unexplained links and quoted posts do not select an arbitrary article. Missing publisher captures remain missing; browsing does not trigger collection, and Reddit cards still open their discussions. See publisher excerpt tests and cross-feed browser tests.

Hacker News stories, articles and discussions

Hacker News uses hckrnews’ UTC front-page history and daily score cutoffs, with Top 50% as the default. Cards show the actual submission time from the official HN API (or the archive’s submission time), never a guessed front-page time. Unavailable metadata remains explicitly unknown. Normal collection fetches story metadata and linked publisher pages; comments require a separate detailed pass.

Optional RSS collection uses native HN discussion IDs and preserves the raw feed entry. It can run alone or alongside the archive/API adapter. Composed snapshots retain previously captured rankings, story bodies and comments; adapters share publisher fetches within a source run. Network failures leave the source partial when another adapter succeeds. RSS-only items have no inferred daily ranking or scores and are available under All. RSS does not reconstruct front-page arrival times or historical days. See RSS tests and composition tests.

Detailed capture now defaults to comment_limit = "all" with no overall thread timeout. Item requests have a 30-second timeout, at most three attempts, bounded concurrency, and shared host limits. Previously capped threads are retried first. Failures preserve saved comments and remain partial; complete means every reachable branch observed during that run was fetched, not that a growing live thread will never acquire new comments. The API no longer truncates at 500.

Publisher pages use the existing article extraction and recovery rules. A denial, challenge or unsupported document remains an unavailable capture. Public paywall excerpts may remain excerpts; collection does not imply subscriber access. Original publisher dates/bylines are extracted when present. Readable existing original-source captures are reused, rather than overwritten by an aggregator’s title or body.

Resolved HN links open the collected destination; All shows its original source with incoming Links. HN cards without submission content now reuse the saved publisher excerpt; submissions with their own text keep it. This is a display fallback and does not replace stored HN content or fetch missing articles. Empty or unavailable publisher content still has no excerpt. Reader link tests and browser tests cover this behavior. RSS aggregator copies, including existing Lobsters records, remain references to a separately collected publisher record. Ask/Tell HN citations do not replace their containing post. Known platform post IDs and observed redirects resolve aliases; ambiguous URLs are kept separate. Links are tied to processed versions and content hashes, so unprocessed captures and reprocessing cannot expose stale link relationships. Malformed legacy anchor URLs are skipped without preventing startup or indexing other links in the same post. Incoming lists show up to 50 references with the full count when more exist; this display limit does not apply to comments.

See configuration and collection, collector tests, article tests, and link identity tests.

YouTube feed discovery

YouTube collection saves feed appearances and their original renderers, with video downloads off by default. Shelf headings, rank and playlist identity come from YouTube; country codes record the configured regional context and capture times record when DANK observed the listing. Separate discovery records preserve repeated appearances across feeds and runs. Descriptions and comments are not collected; video-detail collection is a deferred idea. Feed snapshots are separate from caption captures. Live and upcoming broadcasts are excluded.

The browser follows Hype's regional Top 100 → View all link, expands News sections and uses bounded scrolling. The UK playlist fallback is used only for the GB region. Source layouts, regional availability and recommendation order can change; a successful bounded run is not a complete archive of an infinite feed. Unrecognized/empty or failed feeds increment failure counters and allow later feeds to continue. Already persisted appearances remain available. Channel latest-video listings stop after consecutive known items; this heuristic does not apply to ranked or personalized feeds.

Signed-in homepage Recommendations uses yt-dlp only, with a 30-second listing budget and 2,000-renderer guard. Chromium verifies the existing session and supplies cookies in memory; no browser recommendation scrolling or fallback runs during normal collection. Browser startup and authentication precede this budget; media, captions and indexing follow it. The comparison command still supports explicitly comparing browser and yt-dlp captures.

The isolated yt-dlp worker observes original initial/continuation responses before flattening, so Shorts and community posts remain available. Shelf ancestry and exclusion flags follow observed continuation tokens; an unknown token stops that method as incomplete rather than guessing the shelf. Whole news/Playables/ad subtrees are pruned before normalization, and normalization rejects live/upcoming videos. Ordinary news or gaming recommendations outside excluded shelves remain eligible. Changed renderer shapes, shelf headings or pinned extractor hooks may still require updates.

The reader's Recommendations feed spans all saved homepage captures, deduplicating each item by its latest observation and ordering newest first. The first page freezes the latest persisted capture time; cursors carry that same cutoff separately from their last-item position. Applying it before aggregation keeps older item versions available even if a refresh recommends them again while browsing. Sidebar totals and Channels use the same history. Hype/News keep their latest-run scope. Historical homepage cursors from the old scope are invalidated; reloading starts a consistent archive view.

The reader's All view paginates Videos, Shorts and Posts independently in 500-item batches with explicit Show more controls. Format tabs and the recommendation-derived Channels view use infinite scrolling. Channel counts and three-item previews use the selected feed/search, with cursor-bound snapshots and count/identity ordering. Configured channels only resolves current configured feeds through captured channel IDs and URLs before grouping or pagination; historical unconfigured feeds do not qualify. Missing channel identities remain explicit rather than inferred from display names. See channel API regressions and reader acceptance tests.

The output contains one row per video/post ID and one set of asset references, in yt-dlp order. An explicit Shorts renderer takes precedence over a generic video classification. recommendation_captures preserves original renderers, shelves and source positions, with collection_methods = ["yt_dlp"] on new captures. Failed or timed-out extraction retains partial results; cancellation kills and reaps the worker. Flat entries lacking shelf provenance are not used. See method regressions, worker cleanup and browser comparison exclusions.

Live merged capture, 2026-10-23: Chromium returned 378 identities and the raw yt-dlp path returned 369, with 273 shared. Both completed; deduplication produced 474 items (454 videos, 20 Shorts, no posts) in 26.3 seconds of listing work, or 34.6 seconds including browser startup/session verification/cleanup. Accepted observations retained only ordinary-grid and Shorts shelf context; news and Playables subtrees were excluded. This is one measured account snapshot, not a guarantee that every future homepage finishes before the deadline. A normal reader refresh then saved 482 rows / 482 distinct IDs (461 videos, 21 Shorts): 122 shared, 244 browser-only and 116 yt-dlp-only. Both listings completed in 25.6 seconds, with zero excluded shelf observations in storage. The overall refresh was partial because one thumbnail returned 404 and one caption request failed; all merged posts remained available.

Google sign-in uses one 1Password item read per attempt and generates any requested authenticator code in memory. A successful callback is insufficient: login must authenticate a clean homepage and a new browser instance. A refused vault request stops the attempt; collection never falls back to signed-out recommendations. Native profile cookies are checked first. Only a complete, definitively signed-out homepage permits one cache restore per browser session; a fresh document must then authenticate. Loading or uncertain states trigger neither cookie injection nor vault access. Authentication failure stays local to YouTube collection. The private cache is refreshed only after verification, with page scripts stopped; invalid/expired caches preserve the saved file. See session cache tests and login checks.

Restart verification, 2026-10-23: web login initially worked only while its browser stayed open. Reopening retained YouTube cookies but removed Google account cookies and invalidated the web session. Disabling Chromium's own browser-account sign-in for the isolated YouTube browser fixed this: a fresh login survived reopening and a separate one-off container check, with no new vault read. Login, collection and comparison use the same --allow-browser-signin=false setting; Google website sign-in remains enabled. Chromium's account consistency implementation recognizes this setting. This does not prevent ordinary server-side expiry or future Google verification challenges.

The paired comparison keeps yt-dlp's flat results separate: shelf context and text-only posts are unavailable, and Shorts shelves may be omitted. Reports include raw/unique category totals, partial outcomes, start skew and video-ID overlap. A matched cookie snapshot and close starts reduce timing differences but cannot force YouTube to return identical recommendations. See comparison tests.

Live paired comparison, 2026-10-23: starts were 0.000209 seconds apart. The browser reached an observed end in 27.716 seconds with 322 unique items (304 videos, 18 Shorts, no posts). yt-dlp completed in 5.939 seconds with 315 unique videos plus 27 playlist entries; it omitted all 18 browser Shorts. 209 video identities overlapped. A fresh-process repeat after the session fix reached the browser end in 23.688 seconds with 251 videos and 18 Shorts; yt-dlp returned 251 videos plus 19 playlist entries in 5.711 seconds. Starts were 0.000163 seconds apart, and 188 video identities overlapped. These are request-specific captures, not proof that either method is a strict superset. The regional recommended-feed URL requires yt-dlp's scoped no-youtube-channel-redirect compatibility option; without it, the extractor's exact-URL guard rejects the home response.

The runner can import a captured browser comparison without collecting again or downloading media/captions. Original order, timestamps and source payloads are retained, while yt-dlp rows are excluded. Reimporting a completed or partial snapshot is idempotent. Interrupted imports require inspection before retry because raw posts are append-only. The comparison command remains read-only. See import regressions.

Regional playlist freshness remains unverified across days. The US playlist was identified through a US VPN and returned 100 US Hype items on 2026-10-17; GB uses discovered View all. Unchanged snapshots alone do not establish staleness, and changing region is not a verified country-switch mechanism. The multi-day verification task owns the follow-up; saved IDs, ranks and capture times provide the evidence.

Signed-out runs use a temporary profile; optional signed-in runs use YouTube's own saved profile and require a valid session. Neither shares X's or Reddit's browser. The reader's From your feeds index recognizes explicit YouTube links in saved source data. It does not resolve opaque shorteners such as t.co unless the original capture includes the expanded destination.

See YouTube settings, runner regressions and session regressions.

Live verification, 2026-10-17: signed-out Docker run 08b7d6c0-c6a1-4dcb-af6b-5db8eff0716b saved 715 appearances / 602 unique items from UK Hype, the three default News tabs, and Channel 4 News videos/Shorts. All thumbnail references were saved with zero failures; video downloads remained off. Repeat run 5d17dbdc-0617-4afc-ba00-356500ecf303 stopped at Hype's 100 items without scrolling and at five known channel videos, completing in 20.2 seconds. Reader checks reconciled all 100 Hype ranks across pages and channel-wide counts across feeds. The existing collection supplied 430 distinct linked videos.

Attribution of videos linked from other feeds

The reader uses a separate evidence index rebuilt from saved post bodies, explicit article scopes, and identifiable comments. Each occurrence retains its excerpt, matched destination, capture time and post/comment permalink. Multiple captures collapse by video and occurrence identity; older mentions survive a later capture that omits comments. Counts, search and pagination use this same index. Missing YouTube titles use a neutral video-ID label until public YouTube preview metadata is collected; source headlines are never promoted to titles.

The previous index searched entire payloads and attributed every match to the parent post. This included navigation, recommendations, quoted X posts, Reddit crossposts and parent articles archived alongside comment-feed entries. These are not accepted as direct links from the collected post. Identified RSS-page comments remain comment mentions; unknown page layouts or comments without stable identities are conservatively omitted. This verifies archived evidence, not whether an external link is still available or a comment is still public.

Startup backfills the evidence index in bounded batches before serving the reader; an interrupted backfill can retry without duplicating references. New captures and processed articles add evidence when persisted. Publisher pages collected from aggregator links (linked_article) use the same article/comment attribution rules as RSS captures. Existing version-2 indexes backfill only those publisher records on upgrade; fresh indexes still scan all supported sources. The old index and original source archive are retained for comparison. A read-only audit is available with uv run python -m dank.web.reader_youtube_index --output audit.json (add --config when needed). It reports verified/excluded legacy associations, their source excerpts, and the accepted evidence.

Archive audit, 2026-10-17: of 485 legacy video-to-post associations, 465 were supported by attributable content and 20 were excluded: seven parent articles attached to comment-feed entries, seven nested quotes/crossposts, five navigation/recommendation links and one secondary video-description link. The accepted archive contains 449 post links and 17 distinct comment mentions across 437 videos. No raw source captures were removed.

Covered by extraction tests, migration tests, API tests and reader browser tests.

Linked video preview metadata

Linked videos now have a canonical metadata archive separate from recommendation feed appearances. Public YouTube oEmbed responses provide the real title, channel attribution and thumbnail; original responses and every attempt are saved in youtube_video_metadata. The collector does not download media or claim to have collected descriptions, transcripts or comments. Channel handles do not establish a channel ID when the provider does not return one.

Normal collection enriches up to 100 due, verified linked videos with two workers under the shared HTTP limiter. --youtube-metadata runs that queue without other collection; repeat --youtube-video-id VIDEO_ID to limit it to known videos. Reader collection uses an explicit same-origin JSON POST, validates that the video already exists in saved evidence or feed discoveries, serializes concurrent requests, and limits request and response sizes. GET requests remain read-only.

Successes are cached for seven days. Network, malformed-response and transient HTTP failures retry after one hour; unavailable/private responses (401, 403, 404, 410) retry after one day. These are local retry policies, not guarantees that a video is deleted or private. The latest attempt is shown separately from last successful capture availability. Failed refreshes preserve prior title/thumbnail metadata; genuine selected-feed titles remain tied to their capture. The queue is intentionally bounded; large backlogs need subsequent runs. Preview thumbnails use the provider URL and can become unavailable independently of saved metadata.

Tests cover provider-payload retention, malicious URLs, malformed/oversized and unavailable responses, retry caching, duplicate requests, provenance preservation and source-title isolation in collector tests and API tests.

YouTube transcripts

Existing public captions are collected for up to 25 videos encountered in the current scrape, and on first opening a known video without saved captions. No archive-wide transcript backfill runs. Successful captions are stored with original caption JSON, track metadata, language, origin and timing; successes are reused indefinitely. Missing tracks wait one day, failed retrieval waits one hour. Read-only transcript requests never contact YouTube. On-demand collection is serialized per local web process with a bounded wait; requests from separate scraper/server processes can still overlap. See the cross-process coordination task.

YouTube may block caption discovery or return empty/invalid caption responses. These are failures, not evidence that captions do not exist. Each attempt has a 40-second deadline and size limits; the extractor subprocess is terminated on cancellation. A failed attempt stops further transcript collection in that scrape. Only JSON3 tracks are supported, preferring English then the original language, and creator-provided over automatic within each language. Machine translations are excluded. Automatic captions can contain errors; saved text is not a verified verbatim transcription. Saved captions are not included in semantic search; track that idea under search saved captions.

A failed row's error column names the cause: yt-dlp's own message, an HTTP status, which size limit was exceeded, or every cause when several tasks fail together. The reader doesn't show it; it says only that captions couldn't be fetched. Rows saved before 2026-11-04 may say only unhandled errors in a TaskGroup (1 sub-exception). Those retry after retry_after, when the video is next opened or discovered by a scrape; nothing retries them on a schedule.

Auto-dubbed videos have an automatic caption source for each audio track. yt-dlp's translated_subs skip only covers manual subtitles, so each automatic source is still listed in about 157 languages and seven formats, roughly 0.5 MB per source. About 16 sources overflowed the former 8 MiB metadata limit on every attempt. On 2026-11-04, all 13 videos whose latest row had failed measured 9.6-11.2 MB from 18-21 sources, 98% of the entries being machine translations that are never selected. The limit is now 16 MiB, which fits about 31 sources; a video with more fails on every attempt, and its row names the limit. track_payload keeps the full listing, which is cheap to keep: an 11 MB listing parsed in 18 ms and compressed to 91 KB with the column's ZSTD codec.

YouTube caption request failed (HTTP 429) with a Sorry... page is a block on the whole IP address, not a problem with that video. One on 2026-11-04 followed about 30 lookups and 12 caption downloads in 20 minutes from a manual check. It lasted at least 50 minutes and had cleared 8 hours later.

Validation includes track choice, parsing, negative caching, current-run scoping, read-only access, first-open collection and desktop/mobile rendering. A live 2026-10-19 check retrieved 19 creator-caption segments for L_6pAVHYoFM, and a 2026-11-04 check retrieved 2,065 automatic-caption segments for the 18-source eqYqUAdjsL0; these establish those samples, not universal availability. See collector tests, API tests and reader tests.

YouTube channel identity and totals

Channel identity comes from the channel page metadata/header, never from recommended videos. Avatar caching uses the same bounded image download and signature checks as Reddit icons; a failed or disabled refresh preserves an existing cached icon. Displayed channel totals can be rounded by YouTube; missing totals remain unknown. Collected channel counts include unique videos and Shorts across feeds, excluding community posts. Existing channels gain this metadata on their next collection. After first-visit consent, collection waits up to five seconds for delayed channel initial data. A public Channel 4 News check on 2026-10-17 captured its avatar and displayed 26k videos total. See channel metadata tests and source API tests.

Thumbnail recovery and observed versions

Title/image combination history now records the listing title and its capture time with each fetched thumbnail. Multiple listing captures sharing an image URL retain their own metadata. Pair dates are aggregated per image and title; visually grouped encodings merge dates only for identical titles. Historical matches use an exact URL in one recorded scrape, reject conflicting titles or overlapping runs, and are labelled as matched. Fallback images and records outside a known run remain unpaired. Read-only ClickHouse cases verify recurrence, reused URLs, conflicting titles, fallback images and exact date ranges; tests cover capture metadata, pair grouping and per-image UI captions.

Captured thumbnail URLs can expire or return 404 while another rendition for that video remains available. Collection now tries bounded same-video fallback URLs on 404, retaining the captured URL and the URL actually fetched. Failed candidates are not stored as images; a successful fallback counts as recovery. --retry-youtube-thumbnails repairs saved missing images without recrawling feeds; repeat --youtube-thumbnail-id VIDEO_ID to select videos. This also adopts their older cached files without changing the originals.

Thumbnails are re-fetched on later observations, even if their URL is unchanged. Immutable files use content hashes; actual image formats determine extensions (including AVIF delivered at a .jpg URL). A conservative perceptual comparison groups near-identical renditions for display while retaining every original. Small edits can be grouped by this heuristic; it does not prove two captures came from the same experiment. The reader exposes the latest eight visual versions from the latest 64 distinct content hashes, and 20 distinct observed titles, with first/last capture dates. These are display limits, not deletion rules. Old unvisited videos and unobserved A/B-test versions cannot gain history without another capture. Known-overlap stopping still refreshes encountered videos before ending the channel pass.

Live verification on 2026-10-17 recovered all six missing thumbnail URLs with five sddefault.jpg images and one hqdefault.jpg. Existing captures of bVRiUa04bHE yielded two distinct images and two titles while remaining one video; Chrome verified hover crossfades without playback at desktop and mobile widths.

Regression coverage checks 404 recovery, same-URL changed bytes, immutable storage, resolution deduplication, dated title history and stable video identity. Browser coverage checks hover/focus crossfades, image failures, static reduced motion and desktop/mobile history views. See YouTube guide, reader tests and reader API tests.

Reddit community total estimates

Community headings use Arctic Shift's archived post count as an estimate, marked ≈, rather than claiming a complete Reddit total. Hover text identifies Arctic Shift, its count update timestamp and DANK's retrieval time. Normal Reddit scrapes refresh the metadata once per configured community per run, independently of media settings; browsing and detail-only refreshes make no archive requests. Counts can lag or differ from currently visible Reddit posts. Failed, missing or invalid responses leave the previous estimate and dates intact and do not stop listing collection; fetch/parse failures remain visible in run history. An older archive update cannot replace a newer saved one. Communities without a saved estimate keep their collected-only heading. See collection details, enrichment tests and reader acceptance tests.

Publisher recovery

The follow-up to run b34b70fc-18eb-4ddf-b48c-e6d5eb0b4b73 identified several recoverable request and media failures:

HTTP requests now identify DANK and its project URL. The generic Python User-Agent was rejected by Ars Technica, a Microsoft article and Wikimedia image delivery. Wikimedia's published policy requires a descriptive client identifier and contact information.
A feed explicitly linking back to its own page with Read full article is treated as an excerpt even inside content:encoded. When a page is available, the processor uses its article body; ordinary full feeds remain preferred and failed fetches retain their excerpt. This repairs Ars Technica's short previews after a successful page recovery, including articles whose canonical URL changed. All 68 saved Ars entries were reprocessed with this correction.
RSS media requests carry the source page's origin as their referrer. This restores the shkspr.mobi favicon images without sending article paths, query strings or credentials. HTTPS origins are not sent to HTTP media. Extensionless WebP images are also recognized during processing, so these recovered icons use their local files in the reader.
Internet Archive storage-node links are normalized to the stable /download/<identifier>/<filename> address in collection and reader HTML. The old ia600209.us.archive.org host was unreachable; the canonical URL works. Its 16,821,362-byte video exceeds the configured 10 MiB limit and is correctly skipped, rather than treated as a connection failure.
Automatically discovered and cached WordPress comment feeds are excluded from article collection. Explicit feed_urls still take precedence. This removes Order Order's unrelated, rate-limited comments feed from normal runs.
Etsy engineering articles, Netflix Tech Blog and The Register can recover HTTP 403 or recognized verification pages through a separate RSS browser. Recovery preserves the browser's document response and final URL, records page_fetch_method = "browser", and counts failures only when recovery fails. Each attempt has a 20-second budget and at most one retry. Repeated failures suspend browser recovery for that host for the rest of the run. HTTP 429 retries retain their existing pacing; subscriber pages and other hosts do not receive this fallback. X, Reddit and YouTube browsers remain independent.

Real access problems remain visible: Niche Gamer's feed still presents a Cloudflare verification screen after a bounded browser check; some Etsy pages request interactive verification; LWN explicitly requires a subscription for restricted articles. Two diagrams in Microsoft's MSTest article return 404. The stormatics.tech / dbadevops.com destinations initially closed connections in HTTP and Chromium; both original article URLs recovered on the final rerun. No verification screen, subscription notice or missing-file response is saved as a successful article or silently reclassified as an intentional skip.

See recovery regressions and browser recovery.

Live verification, 2026-10-17: rerun 3bb45aab-d1bd-4799-ad0a-3ee0dce6ae4a selected all 13 affected source entries: nine completed, three remained partial, and Niche Gamer remained blocked. It retained 202 post captures; media failures fell from 12 to the two missing Microsoft diagrams. The original failed-page set contained 70 article URLs; 44 recovered, while Etsy's 20 and LWN's six remained inaccessible. A separate saved-page retry (16a5ec3f-c06c-49c4-b090-b04b1317624e) recovered 81 older captures outside the current feeds, leaving five older Register captures incomplete after bounded recovery. Original payloads remain available.

Full-scrape check: 2026-10-17

On 2026-10-17, Docker run 19bb2610-7163-4591-ba27-f409be8eaa69 used 851e95a and the existing 55-source configuration. It exited normally after 6m25s, with outcome partial: 43 sources completed, 10 were partial, and Niche Gamer and The Register failed. X and Reddit both completed. The archived log is logs/scrape/2026-10-17T14-12-27Z-58404.log.

All 2,456 post captures plus 25 ID-only references were saved: 2,481 raw rows, independently reconciled against run/source counters and account logs. The post captures represent 2,440 distinct posts, not necessarily new posts. References remain in raw storage and do not inflate the post totals. Of their 25 IDs, 23 also have full content in this run; two are reference-only. All 2,440 distinct posts and all 24 available YouTube files were processed and independently checked against their latest captures; the reader remained available.

Source	Post captures	Distinct posts	ID-only references	Result
RSS	806	806	0	Some feeds, articles and media remained unavailable.
X	1,063	1,047	25	All 36 account timelines completed; one browser timeout recovered on retry.
Reddit	587	587	0	All 15 communities completed, including 27 LLMDevs posts; two were explicitly empty.

X again timed out for 30s at account 21, lexfridman. Restarting with the saved session collected 32 posts on retry and all later accounts completed. Reddit took 3m01s; DomainDrivenDesign and ExperiencedDevs had explicit empty Top/day listings, while other communities yielded 1–100 posts. Both browser sources had zero final fetch, parse, rate-limit or media failures. Normal runs deferred individual discussion refreshes as intended; X saved no conversation snapshots.

YouTube retained 24 available MP4s from 52 references; the other 28 were size-excluded, with no unexplained missing files or download failures. All 24 files were cached from earlier runs. FFprobe confirmed audio/video streams and the unchanged 10 MiB cap for every file (largest: 10,229,642 bytes); FFmpeg fully decoded both streams of the three largest files without errors. One extractor warning reported unavailable formats for EEUdslTwYZ8; the final outcome was an explicit no-fitting-format size skip. This pass verified cache reuse, size classification and existing files; the fresh Docker download/playback evidence is recorded below.

Overall media: 51 new, 3,515 cached, 582 skipped, 11 failed. The same eleven publisher-media failures recurred on Wikimedia, Microsoft, shkspr.mobi and Internet Archive. All 63 fetch failures were RSS-related. The four HTTP 429 responses were attempts on one Order Order article. The Register's feed now returned a verification page, preventing collection of its articles; Niche Gamer's feeds remained blocked by 403 responses. This is historical evidence; later publisher repairs are recorded above. Remaining recovery, media and progress work is tracked in collection reliability.

Full-scrape improvement tasks

This section retains the original incident and completed fixes. Outstanding actions are tracked only in collection reliability; these historical results do not reopen already-shipped work.

These follow-ups came from run 3024d25d-d50e-4cde-be87-4b8e70224e22 on 2026-10-17 (Europe/London), using commit 0e0de9a. Its archived log is logs/scrape/2026-10-16T23-41-08Z-50060.log (filenames use UTC). The run used all 55 configured sources, 36 X accounts and 15 Reddit communities. It exited with RedditScrapeError after 6m39s, with 1,420 raw records saved. The saved rows were independently counted in ClickHouse within the run's time window; these are captures, not necessarily previously unseen posts.

Source	Saved records	Distinct post IDs	Outcome
RSS	826	826	Some feeds/articles/media failed; retained records remain available.
X	594	581	Partial: the 21st account timed out; 15 were not attempted.
Reddit	0	0	Failed on the 12th community; 32 queued posts were never flushed.

The source outcomes were 27 completed, 26 partial and 2 failed. The browser timeout prevented an indefinite X wait, and Reddit ran independently, but this run did not establish reliable collection of either browser source. Media counters reported 389 new files, 2,508 cached, 154 failed and 183 intentionally skipped. All 55 saved YouTube references lacked a local file; logs contained errors for 53 distinct YouTube IDs. Size-limit rejections produced 91 warnings for 85 distinct URLs and are included in failure counts.

What the reader tells us

There are currently two collection-quality pills:

Indicator	Trigger	Reader behavior and limits
Not collected	The sanitized article body has neither non-whitespace text nor an image/audio/video/source element with a `src`.	Shows “Full text not collected” and an original-source link. Opening it does not mark it read. This describes the output, not whether scraping or extraction caused the loss.
Incomplete scrape	Content is available, but its RSS capture records a failed page fetch or matches the known Register verification screen.	Keeps available content readable and warns that text, formatting or media may be missing. It does not measure how much is missing.
No quality pill	Content is available without either recorded failure signal.	This is not proof that the article is complete. Summaries, unknown challenge screens, wrong-body extraction and missing media can escape detection.

The incomplete check matches the capture behind the displayed processed article, including its collection timestamp. A new successful scrape clears it after processing, not merely after downloading. Recovering formatting from an older saved page can leave the newer capture's failure warning in place. An empty body with a failed-capture flag still shows Not collected.

“Article/Post” identifies content type; “Read” is reading state. Neither is a collection outcome. API/loading errors have retry messages, and image-load failures have a placeholder. These are separate from the two quality pills.

Implementation: content check, card precedence, capture lookup. Coverage: reader journeys and API tests.

Known source cases

The configured source can differ from the publisher being fetched. For example, the Microsoft article below was collected through lobste.rs; source filters and recovery commands use that configured source identity.

Source / publisher	Observed problem	Current handling	Status / remaining gap
`arstechnica.com`	On 2026-10-15, “Fridge hack costing millions and sparking fires stemmed from poor security” returned HTTP 403. The retained feed supplied five paragraphs, with no article-page HTML.	Retained failure metadata produces Incomplete scrape while the feed text stays readable. Saved failed pages can be retried and then processed. Fetch handling, retry tests.	Mitigated. The badge is evidence of a failed capture, not confirmation that those five paragraphs are the whole article. A retry can still be denied.
`www.theregister.com`	HTTP 403 responses; an HTTP 200 “Are we human?” verification shell; full feed text flattened into one block.	The known verification shell counts as a failure, including older saved shells. Matching article pages restore paragraphs, inline markup and supported figures; processing can reuse an older matching capture. Challenge detection, structure recovery, tests.	Mitigated. The reported Metropolitan Police article was restored to 28 paragraphs on 2026-10-15. Other captures can remain blocked. Detection recognizes this particular verification screen, not every anti-bot page.
`techcommunity.microsoft.com`, collected through `lobste.rs`	The “Microsoft Defender for Linux” capture had a loading shell in its HTML body while the article lived in embedded page data. Ordinary extraction produced no readable text.	The processor reads the matching Khoros blog message from saved Next.js JSON when no readable article container exists. Empty/script-only containers permit recovery; rendered text stays preferred. Unrelated cache entries, comments and drafts are excluded, scripts are not executed, and normal sanitization still applies. Extractor, regression tests.	Handled for this layout. Merged and reprocessed on 2026-10-15: all 587 words, 12 paragraphs, six headings, four lists and seven links verified on desktop/mobile; the image was separately collected and verified from the local cache. Original raw capture and post metadata stayed unchanged. Other client-rendered layouts remain unsupported; reprocessing does not automatically collect media found only in embedded JSON.
BBC domains	Homepage discovery needs an explicit source-specific fallback.	The scraper supplies built-in BBC RSS URLs; configured `feed_urls` bypass automatic discovery. Discovery behavior, tests.	Handled for the supplied feeds. This does not guarantee their availability or coverage of every BBC section.
`x.com`	Login/session challenges or confirmation requirements can stop timeline collection.	Session-only collection; `make x-login` checks the saved session first and reports reuse when valid, skipping 1Password access and Docker desktop startup. New sign-in uses the same profile as collection. Expired or challenged sessions count as a fetch failure and stop X collection. X guide, collector.	Mitigated. The RSS incomplete flag does not describe X collection failures. Use run history and logs. Optional 1Password autofill fills visible X fields once without submitting them; manual sign-in remains available when the CLI is missing or access is denied. Complete verification in the sign-in browser. A visible temporary-login-limit message stops sign-in immediately; do not retry until X clears the restriction. X may still reject automated browsers or change its authenticated navigation markers. Session expiry, cleanup and profile locking are covered in `tests/scrape/test_x_session.py` and `tests/scrape/test_browser_session.py`; browser defaults and limit handling are tested locally without live login attempts. Reuse is reported only after detecting authenticated navigation. X may still require verification or temporarily limit login attempts. The homepage and account pass share one tab. Account switches open search with `/`, keep the combobox focused, press Down until the exact account is highlighted, verify its avatar handle in the owned popup, and press Enter without a document reload; missing suggestions or a profile that fails to open time out and skip to the next account without reloading the tab. Timeline scrolling jumps to the loaded bottom. Ready responses drain immediately; empty queues use 0.1-second waits with configured idle backoff. The homepage only allows 1.5 seconds for its first response. Login prompt checks no longer poll on every scroll. Mouse activation and focusing the nested result button failed in operator testing, so keyboard selection now stays on the search input. All sources use the existing discovery/save loop and independent asset fetching. The asset consumer drains the bounded handoff into a media backlog, so downloads cannot pause browser navigation; post writes still finish before discovery advances. Media batches fetch in order under shared HTTP/media limits. The in-memory media backlog has no separate byte cap, and cancellation can leave saved posts with undownloaded assets. Browser sessions remain open through source discovery and fetching; each source worker waits for its media before moving to another source. Regression evidence includes blocked media tests for X and generic sources, browser session lifetime, durable post writes and task cleanup in `tests/scrape/test_scrape_history.py`, homepage tab reuse in `tests/scrape/test_x_home.py`, and immediate response drains in `tests/scrape/test_x_collection.py`. Search selectors and suggestion availability depend on X; faster scrolling remains a bounded sample and does not prove complete history. Navigation matching, missing-result timeouts and pacing are covered in `tests/scrape/test_x_navigation.py`; browser recovery is covered in `tests/scrape/test_x_collection.py`. The November 4 log showed the first account timing out after homepage discovery, without identifying the search stage. Search now logs each stage, activates the tab and sends explicit native Down/Enter events: zendriver's built-in Enter encoded `key` as a carriage return. Live popup avatar attributes replace descendant-text snapshots for suggestion matching. An isolated headless Thorium check verified trusted, focused Down, Down, Enter navigates a local combobox fixture with the supplied popup structure and an initially missing active-descendant ID; wire-event regression tests cover the key names. The subsequent November 4 run navigated some accounts successfully but logged Enter followed by timeouts for RupertLowe10 and Sargon_of_Akkad. Selection waits for the expanded popup to contain the requested account, observes each selected-option change before another Down, and requires the exact option's `aria-selected="true"` marker before Enter. An already selected requested account is accepted without another Down. The operator's initial and selected HTML snapshots both retain a zero-percent progress bar; waiting for its removal was incorrect. The 11:32 November 4 log confirms that check stalled at popup readiness before any Down. Progress-bar presence no longer gates selection. The route and the main profile header's handle must both match before scrolling. Following further operator-observed input races, each Down/Enter now has a 250 ms pacing pause in addition to the readiness checks. Enter selection is revalidated after that pause; polling still waits for observed state changes. Unit tests cover delayed selection, selected-option readiness, an already selected account, progress-bar-independent selectors and stale profile headers. Four isolated Thorium checks passed with a persistent zero-percent progress bar, immediate or delayed highlight/profile updates, and initially unselected or already selected accounts. The earlier fixture incorrectly removed the progress bar and did not establish behavior for the operator's markup. The 11:41 log showed Enter followed by a timeout for dickmasterson, then successful navigation after restart. Before Enter, the collector now confirms the exact selected option and matching active-descendant ID remain unchanged across two animation frames; a re-rendered or mismatched option cannot authorize Enter. Existing search text is cleared and emptiness verified before typing. Account timeouts and unexpected account errors preserve saved batches and move to the next account in the same tab without retry/reset, keeping browser asset requests alive. Discussion timeout/unavailable results likewise continue without resetting. Connection/protocol errors still use bounded browser recovery; sign-in requirements stop X collection. Tests cover reused tabs, skipped accounts, retained batches, clear-input readiness and deferred Enter. This follow-up still needs signed-in X verification; live throughput remains unbenchmarked.
`reddit.com`	Community listings may be blocked, empty, truncated, or stop loading more posts.	Browser collection preserves available listing content, excludes ads and other communities, and stops at configured post/scroll/stall limits. Top for the past day is the default. Access failures are reported explicitly. Reddit guide, collector tests.	Bounded collection. Individual post bodies/media and up to 50 comments including replies (Best by default) are now captured with configurable limits and sort. Body extraction prefers Reddit's semantic article fragment so expand/collapse controls stay out of reader text; older markup falls back to its body slot. Blocked detail requests retain the feed post; comments are explicitly captured, disabled, or failed, while historical posts remain not captured. Rendered-thread fallback can return fewer than the limit because replies are collapsed or loading stops; the reader indicates more comments. Failed detail refreshes preserve the last successful body/media. Failed or disabled comment refreshes preserve the archived discussion with its original capture date and latest attempt state; successful empty captures replace it. Native video is remote by default; optional downloads use yt-dlp to assemble audio/video, gated by both Reddit and global media settings. Remote links can expire or be blocked. A completed run does not prove complete daily coverage. Detail capture tests. The RSS incomplete flag and `--retry-failed-pages` do not apply to Reddit. Community icons are discovered from the public header; layout changes or a missing custom icon use initials or the previous cache. Invalid image responses are discarded so a later scrape can retry. Flair choices reflect collected posts and their latest raw snapshots, not the full community flair catalogue. Icon cache tests, flair tests.

Shared handling and gaps

These are implemented safeguards or known limitations across sources. Generic test fixtures establish behavior, not an incident at a particular publisher. Related work is tracked under feed/article recovery, media outcomes, and processing outcomes.

Problem	Handling today	Current limit
Graph connections can lose provenance or misattribute links in page chrome and embedded foreign posts	A separate evidence index extracts owned post text, structured HN/Reddit comments with verified ancestry, scoped publisher bodies and identifiable article comments. X quotes, recommendation payloads, comment author/reply controls and self-links are excluded. URL aliases resolve collected destinations; publisher path/query meaning is preserved, and only saved YouTube metadata supplies video titles. Raw captures are unchanged.	Saved evidence only. The first startup scans the archive once in 64-row batches; future collection/processing writes evidence at the persistence boundary. Graphs show up to 100 occurrences in each direction with totals; older evidence retains its date and exact lookup. Only processed origins are navigable; newer unprocessed evidence is withheld. Missing raw ownership fields, uncollected comments, unsupported article layouts and X replies not saved as standalone posts remain outside coverage. No semantic, sentiment or topic relationships are inferred. Extraction tests, API and migration tests.
Article fetch fails, times out or returns an empty body	By default the entry is skipped. With `keep_feed_on_fetch_failure = true`, the feed entry is retained with `page_fetch_status: "failed"`. Warnings record the URL and available error detail.	Retention is opt-in. The raw flag does not retain a structured HTTP status or detailed reason; inspect logs for that. `--retry-failed-pages` is manual, not a scheduled retry. RSS guide.
HTTP 429 / host cooldown	RSS/homepage requests have up to three retries with 2/4/8-second backoff; `Retry-After` and shared per-host cooldowns are respected within the 30-second wait limits.	Other HTTP errors and timeouts do not get those same-run retries. Direct-media 429s start a cooldown but that download is not retried in the same run. Limits, tests.
Missing or malformed feeds	Explicit feed URLs can bypass discovery. Fetch/parse failures are counted and logged; a source without usable feeds is reported as skipped or failed according to its counters.	An unavailable or invalid feed yields no entries to retain, and therefore may produce no reader card or pill. A valid empty feed can complete with zero posts. History, runner tests.
Browser profile stays locked after a Docker scrape	Chromium gets up to five seconds to exit normally before driver cleanup, allowing it to remove its singleton links.	Handled for normal shutdown. Two consecutive one-off Docker scrapes on 2026-10-16 reused a profile and left no singleton links while preserving profile data. Crashes or forced termination may still leave a lock; verify that its owner is gone before clearing stale links. Browser lifecycle tests.
X stalls while Reddit waits for the shared browser	X and Reddit use separate browser processes, persistent profiles and per-site locks. X keeps its existing login profile; Reddit starts with its own profile. X browser startup/navigation and scroll steps have 30-second budgets; a timeout closes the owned browser and retries that account once, continuing after a second failure. Sign-in/profile-lock failures and unsafe cleanup still stop X.	Cross-site blocking removed; X driver waits bounded. A Docker run inspected on 2026-10-17 had saved 826 RSS posts and no X posts before X stopped responding and Reddit remained queued. The run was cancelled with saved rows intact. Regression tests verify Reddit persistence while X waits, same-site serialization, separate media cookie paths, timeout reporting and capture-handler cleanup. Runner tests, X tests, concurrency. Both sites still use the configured source-worker limit. Account retries are bounded; there is no overall run deadline. The recurring timeout diagnosis is tracked in the backlog.
X account pages display posts but collection captures none	Capture recognizes `UserOriginalsTimeline` as well as the older account timeline endpoints; it still excludes the personalized home timeline.	Observed endpoint change handled. On 2026-10-17 a saved session in headless Docker received 27 parseable posts from `UserOriginalsTimeline`, which the old capture filter ignored. After the fix, a single-account Docker scrape saved 33 X posts in 6.4 seconds with no fetch failures; media downloads were disabled. No new login was needed. This establishes that headless collection can work with that session, not that X will always accept it. Capture regression tests. Future endpoint or payload changes may need another update.
Large HTTP header fields	Direct HTTP requests allow header fields up to 16 KiB, including redirect responses.	Larger fields still fail. This is a parser-limit fix, not an access-control workaround. Regression tests.
X text repeats, links disappear or attachments are absent	Processing prefers full note text, escapes literal text, preserves paragraphs and expands saved URL entities. Reader cards and detail share post content, link cards and ordered media; matching local files take precedence over captured X URLs. Metadata uses the exact processed capture rather than a newer pending scrape.	Saved content recoverable. The 2026-10-17 `@3blue1brown` example already contained its YouTube destination, card title and thumbnail; stripping trailing short links and requiring downloaded assets hid them. Use the X reprocessing command to rebuild saved records. Remote media can expire; unsupported video formats retain their poster and original-post link. Missing raw metadata falls back to processed text and saved media. Reply, repost, like and view counts describe that same capture; missing or malformed counts remain unknown rather than zero. Conversation snapshots are collected separately with bounded detail browsing; reprocessing alone fetches neither conversations nor missing media. Processing tests, content tests, reader tests.
X, YouTube or Reddit video keeps playing outside the reading view	A shared reader controller pauses the previous player when another starts, and pauses a player when it leaves the viewport or the reader navigates. Pausing preserves position; returning to view requires manual resume. Feed and opened-post players share the latest per-video position within the reader tab, including backward seeks. Reloading clears this navigation state. Reddit cards and opened posts share the same controller, including HLS and verified MP4 fallbacks; delayed loading cannot restart a cancelled player.	Reader behavior covered. YouTube is controlled through its iframe message channel, restricted to the matching frame and origin. The bridge waits for readiness before starting, cancels stale starts, and cleans up listeners and polling. This provider protocol may change; unavailable embeds retain Watch on YouTube. Playback lifecycle tests, browser coverage, navigation handoff tests.
Reddit video URLs repeat or portrait players have large side bars	The reader hides only raw references matching a displayed clip, including archived copies and signed URL variants. Failure restores the reference; descriptive links and unrelated clips stay visible. Player sizing uses poster dimensions before playback, then the video's intrinsic dimensions.	Existing saved posts supported. The API retains the original clip URL beside local media. Missing or incorrect posters are corrected when video metadata loads; streams still load only on demand. Reader tests, URL matching tests.
X conversations contain only part of the replies or thread context	Collection saves bounded, focal-post-validated `TweetDetail` snapshots in a separate `--x-details` pass, or for selected saved posts via `--x-post`. Parent context and replies are archived together; ads, nested quotes and unrelated recommendations are excluded from conversation slots. The reader retains earlier snapshots when a later attempt fails or skips detail capture.	Partial coverage is explicit. Budgets, remaining cursors, hidden branches, missing context and reply-count shortfalls mark snapshots partial. Missing responses, sign-in failures and HTTP failures never claim zero replies or completeness. A live signed-in Docker check on 2026-10-17 saved 22 related posts from a validated detail response and correctly reported a remaining cursor. Capture does not expand hidden branches or download extra reply media; remote media may expire. Conversation guide, collector tests, CLI tests.
Flattened feeds, lost figures/captions, relative URLs	Full plain-text feeds recover structure only from complete ordered text matches. Supported figures between matching sibling blocks are retained. WordPress caption containers become figures; final response URLs and valid HTML base URLs resolve links/media.	Formatted feeds stay authoritative; incomplete matches stay unchanged. Older captures without redirect metadata may need a fresh scrape. These rules do not validate every page layout. Processing behavior, caption tests, redirect tests.
Missing or failed media downloads	Failed direct HTTP downloads remove partial files; absent local files are omitted from processed assets. `files_failed` and `media_skipped` distinguish failures from reference-only, disabled and known size-excluded downloads.	Media failure alone does not trigger Incomplete scrape. An article may remain readable with remote images or unavailable media. Configured size limits can also prevent downloads. Media guide, asset tests.
Extensionless images, observed on Microsoft Community Hub and `shkspr.mobi`	Downloaded PNG and WebP images with no filename type are recognized from their signatures, letting the reader use local copies. Known filename types stay authoritative.	Handled for PNG and WebP. Other unknown formats remain unclassified. Existing asset processing is incremental; already-processed unknown metadata needs a refreshed raw asset. This does not discover or download embedded media automatically. Media guide, classification and reader tests.
Processing fails or produces no usable post	Raw captures remain available for reprocessing. Converters can return no post and be skipped; embedding/database exceptions can stop processing after earlier batches have been saved.	There is no per-article processing-error pill or durable processing-status ledger. The reader may show an older processed version or no record. Use process logs and compare raw/processed timestamps. Processor, tests.

Scrape outcomes such as partial, failed, skipped and cancelled are run/source outcomes, not additional article pills. partial means failures occurred alongside saved records; intentional media skips alone do not make a run partial. A zero command exit or completed run does not prove article completeness. See history semantics.

Choose the recovery path

Identify the failing stage. Check the source/post identity, the raw capture behind the displayed version, and matching scrape/process logs. Record whether the problem is fetching, extraction, processing or media.
Missing/blocked article page: run a scoped scrape --retry-failed-pages, then process the new captures. This works after an entry leaves its feed; it cannot bypass a publisher's refusal or recover an entry never retained.
Usable page already saved: fix/validate extraction and use scoped process --reprocess. Reprocessing alone cannot fetch a missing page. Preserve original raw captures.
Verify the result: compare the extracted text and structure with the saved source, inspect the reader, and check the relevant warning state. Mark an issue handled only after the change is in main and verified.

--domains selects configured sources; the processing age window uses capture time. Native and Docker commands are in the linked guides. Per-run archives are created by make scrape / make process; an ordinary direct invocation does not create those archives.

Keep this register current

When adding a quality indicator, fallback, domain workaround or recovery fix, update the relevant row with its trigger, affected source/example, actual handling, remaining gap, verification date and code/test reference. Add a new row only for a distinct problem. Keep proposed or unmerged fixes visibly separate from shipped handling, and retain useful resolved cases so the reason for a workaround is not lost. Keep credentials and full raw payloads out of this document; link to code/tests and summarize dated observations. Put action checklists and future proposals in Linear, linking back here for evidence. Keep only current behavior and recovery guidance here.

X For You recommendations

Authors groups readable For You posts by saved handle, preserving distinct counts across repeated captures. Search and dates apply before grouping; pagination carries the original capture cutoff, with count/handle tie-breaking. Up to three recent examples reuse the saved-post reader adapter. Unattributed posts are reported separately. An author filter stays within For You and does not turn Refresh into an account crawl. Regression coverage is in tests/web/test_reader_x_authors.py and reader/tests/x-authors.spec.ts.

The signed-in homepage has its own X → For You history. Collection listens only for HomeTimeline, preserves original items and their capture positions, and excludes ads, Following responses and nested quotes from feed membership. Bottom scrolling reaches pagination past tall media cards; collection remains bounded by post/scroll limits and a 90-second budget. Failed or empty captures report failure while retaining previously saved history. Sign-in uses the same private X profile as account collection; an expired session needs make x-login.

Pagination pins the capture cutoff and deduplicates posts across appearances. An account refresh cannot remove earlier homepage membership. Search dates still refer to post publication. Homepage replies are included; conversation collection is separate. Tests cover membership, quotes, ads, modules, errors, pagination and refresh scopes in tests/scrape/test_x_home.py, tests/web/test_reader_x_home.py and reader/tests/x-for-you.spec.ts. X can change its endpoint or payload shape; the history is a saved sample, not every recommendation the site may serve. On 2026-10-25, a signed-in Docker refresh captured 155 recommendations plus quoted/context posts, processed 195 posts, and completed with zero fetch, parse or file failures. Two captures yielded 158 distinct readable recommendations; all six pages and a cursor spanning the refresh were checked for duplicates.

X saved reply relationships

X feeds default to Posts; Posts & replies keeps strict date ordering while showing parent links. Independently saved posts are connected in the opened reader without requiring a discussion capture. This does not collect missing replies: profile timelines can omit them, and a saved parent or child does not prove complete coverage. Unknown reply metadata remains visible in Posts. Related lookups use displayed raw versions and are bounded to 200 posts and 20 ancestor levels; gaps and limits are explicit. Archive dates and capture counts retain their original meaning. Covered by relationship tests and reader acceptance tests.

X embedded quoted posts

On 2026-10-18 the saved Eric Weinstein post 2110938986801938839 contained Haider's quoted post 2110797855879471162 and its video, but the reader omitted the entire quote. Feeds and opened conversations now render one embedded level from the parent snapshot, with a separately saved post as the ID-only fallback. Quoted posts and their media are already discovered by timeline collection; this change exposes those captures without another scrape. Saved local files are preferred, and remote playback still depends on the captured URL remaining available. The example's direct video returned HTTP 403 in Chrome; a targeted retry saved its captured 720p rendition within the existing download limit and verified local playback. Missing/deleted quote content stays explicit; this does not collect all replies or fetch missing quotes when opening the reader. Covered by quote API tests, collection tests and reader tests.

Refresh from the reader

Reader Refresh starts a collection for the current source context or saved post. All-view platform choices and source/tag filters narrow collection. Search text, dates, flairs and read markers remain display filters; they do not change a source's configured collection limits. Individual Reddit/X post refreshes request details and comments explicitly. YouTube video refreshes retry metadata and reuse saved successful transcripts; community-post detail refresh and the computed From your feeds view do not have a standalone collector.

Up to three independent refreshes run in separate child processes, with at most five outstanding requests and duplicate-click reuse. A repeated request joins the active job only while the sources it selects are unchanged, so a refresh after a saved source edit queues a new job. Overlapping source domains and shared browser profiles wait; an unrestricted All refresh runs alone and keeps its place ahead of later requests. Command-line scrapes remain exclusive; reader jobs wait and retry when that lock is held. Public article recovery uses separate temporary browser profiles so Blogs and Hacker News can run together.

Processing uses only the posts saved by that run. The notice distinguishes running and waiting jobs, and Show updates opens the completed job's source view or post instead of reloading an unrelated current feed. Each completed result has its own action. Partial collection and processing failures remain visible. Completion means the attempted collection finished, not that every post on a platform is archived. See concurrency tests for source/profile locking.

Recent job results persist locally. A server restart marks unfinished jobs as interrupted instead of replaying them. The queue and status are local to the running DANK installation; provider blocks, configured collection limits and unavailable original pages still apply. See scope tests and job lifecycle tests.

YouTube playlist snapshots

Cards read view counts and publication-age labels from saved listing renderers, including older snapshots. Missing fields remain unknown; no per-video request is added. Rounded counts and relative ages describe the capture time shown on hover. These changing labels do not invalidate the membership hash or trigger repeated processing of unchanged playlists.
Playlist contents use an isolated yt-dlp worker; browser scrolling remains responsible for saved-library discovery and its Recently added order. The pinned extractor owns HTTP and pagination; a selected-tab/continuation observer retains original occurrence IDs, duplicates, sort settings and raw payloads before flat extraction drops them. No watch pages are extracted.
The extra yt-dlp unavailable-video reload is disabled to preserve the selected visible manifest and sort setting. Visible unavailable placeholders are kept; a header/manifest count gap remains partial and can still update when every continuation finishes. Unknown, repeated or interrupted continuations cannot replace the previous saved snapshot. There is no browser contents fallback.
The signed-in browser session is verified before cookies are passed to either yt-dlp collector through private pipes. Cookies are not written to worker files, arguments or logs. A stale session fails explicitly.
Library ranking is published only after a complete authenticated catalog traversal. A partial catalog keeps the previous order; checking one playlist does not move it to the top. Recently added follows YouTube's captured library order, not collection timestamps. The ranking and sort choice are pinned during pagination; playlists absent from the captured catalog follow it A–Z.
Saved-library discovery uses the separate authenticated YouTube browser session; public configured playlists do not need sign-in. It retains observed cards if catalog traversal fails and never deletes a saved playlist merely because it was absent from a later library listing.
A full ordered flat manifest is required to establish an unchanged hash. Counts and first pages are insufficient. Sorting, duplicates, additions, removals and title changes are covered by collector tests.
Checks preserve the previous snapshot on HTTP errors, repeated/stalled continuations, unknown renderers or time/entry limits. The first partial snapshot is usable; subsequent interrupted checks retain it. A fully exhausted visible listing can still contain fewer entries than YouTube's header reports. Such snapshots update their membership and order while remaining partial, with the count described as entries not shown by YouTube. The cause is not inferred as privacy, deletion or regional availability. Membership and ordinary video details are durable before a new snapshot pointer becomes visible. Changed snapshots do not trigger video downloads or automatic caption collection.
Library work rotates through up to eight oldest/never-checked playlists per run, with a three-minute total budget, a one-minute catalog budget and a 1,000-card discovery guard. Per-playlist limits are two minutes and 10,000 entries; use the individual refresh to target a deferred playlist.
Saved Shorts uses a separate grid layout. Its captured array order is retained only when the playlist header and every Shorts card identify the requested playlist, excluding unrelated recommendation shelves. Original Shorts renderers and occurrence IDs are retained. Other unknown layouts fail rather than become empty snapshots. Unavailable placeholders are kept when present; missing placeholders remain an explicit count gap.
tests/scrape/test_youtube_playlists.py covers order/sort hashes, duplicate occurrences, unavailable entries, full continuation traversal, repeated-token failure, unchanged reuse, partial preservation and pointer publication.
tests/scrape/test_youtube_ytdlp_playlists.py checks the pinned extractor, excludes suggested-video continuations, and covers worker failures and empty snapshots. Live checks on 2026-10-26 matched browser order and hashes for Office (2 entries), Saved Shorts (2) and Artificial Intelligence (1,955). That large listing took 13.3 seconds with yt-dlp versus 28.6 with Chromium; the 15-entry source count gap remained explicit in both captures.

Category training and assignment

Training uses OpenAI Decisions predicate probabilities; calibration must be checked against reviewed examples. Refusals, missing answers and invalid scores fail without writing that example; bounded retries handle transient errors. Completed requests are retained if another request fails. Posts with a blank embedding algorithm are eligible for training, and their examples retain the blank value. Assignment treats algorithm names as metadata, so blank-algorithm training examples remain usable after reprocessing records a named algorithm. Only vector dimensions, category names and the thresholds filter these matches. Reprocessing is required to refresh categories after training or threshold changes. Category description edits do not invalidate saved examples; rename a category when changing its meaning. Training omits zero-confidence rows; an all-zero result still counts toward the post limit. Omitted scores can be requested again in subsequent runs. Concurrent training commands can append duplicate examples. Exact cosine queries preserve threshold semantics but cost a scan per new post. No lossy quantisation or ANN index is enabled. See category behavior and research.

Training now lives in dank.training, with import boundaries enforced by run-linters.sh. Decisions receives only URL, author, title and HTML; vectors remain local. Category definitions occur only in predicate questions. The implicit shared-input cache can reuse identical post evidence across retries or missing-label requests when the matching prefix reaches 1,024 tokens. The short developer instructions alone do not qualify for cross-post reuse. No live cache hits have been measured. Verification covers payload fields, cache-prefix stability and import boundaries in tests/training/ and tests/test_lint_architecture.py, alongside category queries in tests/process/test_categories.py and vector checks in tests/test_categories.py. Category loading and training tests use in-memory storage fakes and run with plain make test, without a ClickHouse server. They check query filters/bindings and processing behavior; they do not execute SQL against ClickHouse. Transport replies are simulated; no live paid OpenAI call was made.

Category training refusals and rate limits

Decisions can return an explicit refusal for individual category questions in an HTTP 200 response. Training omits these scores and persists valid sibling answers, continuing to later posts. Fully refused posts count toward the run limit; refusals are not durable records and can be requested again on a later run. Refusal counts are logged without remote refusal text or post evidence. Invalid evidence and malformed answers are logged and skipped per post, counting toward the invocation limit. Later posts continue. Tests in tests/training/test_runner.py verify this with concurrency set to one so the later request starts after the failure. The response alone does not identify a specific safety policy.

HTTP 429 and server failures follow the bounded three-attempt retry policy, using Retry-After where available (delays capped at 30 seconds). Exhausted retries still stop training after successful concurrent siblings have been persisted. Logs include status, attempt, bounded request ID and retry delay; historical conflated refusal/non-predicate errors cannot establish the exact answer type retroactively. Tests in tests/training/test_decisions.py cover partial/full refusals, safe diagnostics, malformed responses and bounded HTTP retries. No live paid request was used.

Navigation Menu

Repository navigation

known-issues.md

known-issues.md

Known collection and processing issues

Browser protocol compatibility

Reader refresh results

Source edits in config.toml require an app restart

Blocked Reddit collection and cancellation

Linked collection and discovery timing

Blog source icons

Publisher excerpts across feeds

Hacker News stories, articles and discussions

YouTube feed discovery

Attribution of videos linked from other feeds

Linked video preview metadata

YouTube transcripts

YouTube channel identity and totals

Thumbnail recovery and observed versions

Reddit community total estimates

Publisher recovery

Full-scrape check: 2026-10-17

Full-scrape improvement tasks

What the reader tells us

Known source cases

Shared handling and gaps

Choose the recovery path

Keep this register current

X For You recommendations

X saved reply relationships

X embedded quoted posts

Refresh from the reader

YouTube playlist snapshots

Category training and assignment

Category training refusals and rate limits

FilesExpand file tree

known-issues.md

Latest commit

History

known-issues.md

File metadata and controls

Known collection and processing issues

Browser protocol compatibility

Reader refresh results

Source edits in config.toml require an app restart

Blocked Reddit collection and cancellation

Linked collection and discovery timing

Blog source icons

Publisher excerpts across feeds

Hacker News stories, articles and discussions

YouTube feed discovery

Attribution of videos linked from other feeds

Linked video preview metadata

YouTube transcripts

YouTube channel identity and totals

Thumbnail recovery and observed versions

Reddit community total estimates

Publisher recovery

Full-scrape check: 2026-10-17

Full-scrape improvement tasks

What the reader tells us

Known source cases

Shared handling and gaps

Choose the recovery path

Keep this register current

X For You recommendations

X saved reply relationships

X embedded quoted posts

Refresh from the reader

YouTube playlist snapshots

Category training and assignment

Category training refusals and rate limits