DuckDuckGo's HTML search endpoint now requires vqd session tokens,
making server-side search-based URL resolution unreliable. Simplified
the thumbnail pipeline:
- OG scrape only for non-aggregator URLs (direct publisher pages)
- AI image generation via OpenRouter for aggregator articles (Google News)
- Added Image Model settings field in admin UI
- Added thumbnail_local column for AI-generated images
- Updated public/admin display to check thumbnail_local as fallback
- Removed all DuckDuckGo search code (find_article_url_via_search,
extract_ddg_url, resolve_article_for_scraping)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google News RSS stores encrypted article URLs that cannot be resolved
server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace
the failed decode/parse strategies with DuckDuckGo search-based lookup
using article title + publisher name to find actual article pages, then
scrape OG images from those pages.
- Add find_article_url_via_search() using DuckDuckGo HTML endpoint
- Add resolve_article_for_scraping() combining redirect + search fallback
- Mark failed lookups with '_no_image' sentinel to avoid repeated retries
- Backfill button resets sentinels to allow manual re-scraping
- Clean up verbose debug instrumentation from prior attempts
- Update frontend to handle '_no_image' sentinel gracefully
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Temporary debug version that returns query counts, URL resolution results,
and OG scraping outcomes to identify why thumbnail scraping returns 0.
Also adds Google News URL decoder (base64/protobuf approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OG image scraper was using Google News redirect URLs (source_url) directly,
which don't serve og:image tags. Added resolve_scrape_url() to follow redirects
(including meta-refresh and JS redirects) to reach the actual publisher page
before scraping. Also added a backfill AJAX endpoint and admin button to
re-scrape thumbnails for existing articles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google News RSS feeds report every article as coming from
"news.google.com" because the source was extracted from the feed URL
rather than the individual article. This adds a multi-strategy
resolver that extracts the real publisher name via:
1. SimplePie <source> element (Google News provides this)
2. Title suffix parsing ("Headline - Source" format)
3. HTTP HEAD redirect resolution as a fallback
Also adds a one-click "Fix Article Sources" backfill button on the
settings page to correct existing articles in the database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-tier filtering during article fetch: keyword/pattern rules as a fast
first pass, then optional LLM classification for borderline cases via
OpenRouter. Configurable sensitivity (strict/moderate/lenient), editable
keyword lists, blocked domain list, and admin stats display. Includes
30 unit tests covering domain blocking, title/body keywords, URL paths,
edge cases, and prompt structure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
