Temporary debug version that returns query counts, URL resolution results,
and OG scraping outcomes to identify why thumbnail scraping returns 0.
Also adds Google News URL decoder (base64/protobuf approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OG image scraper was using Google News redirect URLs (source_url) directly,
which don't serve og:image tags. Added resolve_scrape_url() to follow redirects
(including meta-refresh and JS redirects) to reach the actual publisher page
before scraping. Also added a backfill AJAX endpoint and admin button to
re-scrape thumbnails for existing articles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google News RSS feeds report every article as coming from
"news.google.com" because the source was extracted from the feed URL
rather than the individual article. This adds a multi-strategy
resolver that extracts the real publisher name via:
1. SimplePie <source> element (Google News provides this)
2. Title suffix parsing ("Headline - Source" format)
3. HTTP HEAD redirect resolution as a fallback
Also adds a one-click "Fix Article Sources" backfill button on the
settings page to correct existing articles in the database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-tier filtering during article fetch: keyword/pattern rules as a fast
first pass, then optional LLM classification for borderline cases via
OpenRouter. Configurable sensitivity (strict/moderate/lenient), editable
keyword lists, blocked domain list, and admin stats display. Includes
30 unit tests covering domain blocking, title/body keywords, URL paths,
edge cases, and prompt structure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
