DuckDuckGo's HTML search endpoint now requires vqd session tokens,
making server-side search-based URL resolution unreliable. Simplified
the thumbnail pipeline:
- OG scrape only for non-aggregator URLs (direct publisher pages)
- AI image generation via OpenRouter for aggregator articles (Google News)
- Added Image Model settings field in admin UI
- Added thumbnail_local column for AI-generated images
- Updated public/admin display to check thumbnail_local as fallback
- Removed all DuckDuckGo search code (find_article_url_via_search,
extract_ddg_url, resolve_article_for_scraping)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google News RSS stores encrypted article URLs that cannot be resolved
server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace
the failed decode/parse strategies with DuckDuckGo search-based lookup
using article title + publisher name to find actual article pages, then
scrape OG images from those pages.
- Add find_article_url_via_search() using DuckDuckGo HTML endpoint
- Add resolve_article_for_scraping() combining redirect + search fallback
- Mark failed lookups with '_no_image' sentinel to avoid repeated retries
- Backfill button resets sentinels to allow manual re-scraping
- Clean up verbose debug instrumentation from prior attempts
- Update frontend to handle '_no_image' sentinel gracefully
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace batchexecute API with direct fetch + HTML parsing strategies:
data-n-au attributes, anchor href extraction, meta refresh, JS redirect.
Add verbose debug output to diagnose server-side response from Google News.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace broken base64 decoding with Google's internal batchexecute API
endpoint to resolve encrypted article IDs to actual publisher URLs.
This is the approach used by known Google News decoder libraries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Temporary debug version that returns query counts, URL resolution results,
and OG scraping outcomes to identify why thumbnail scraping returns 0.
Also adds Google News URL decoder (base64/protobuf approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OG image scraper was using Google News redirect URLs (source_url) directly,
which don't serve og:image tags. Added resolve_scrape_url() to follow redirects
(including meta-refresh and JS redirects) to reach the actual publisher page
before scraping. Also added a backfill AJAX endpoint and admin button to
re-scrape thumbnails for existing articles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google News RSS feeds report every article as coming from
"news.google.com" because the source was extracted from the feed URL
rather than the individual article. This adds a multi-strategy
resolver that extracts the real publisher name via:
1. SimplePie <source> element (Google News provides this)
2. Title suffix parsing ("Headline - Source" format)
3. HTTP HEAD redirect resolution as a fallback
Also adds a one-click "Fix Article Sources" backfill button on the
settings page to correct existing articles in the database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-tier filtering during article fetch: keyword/pattern rules as a fast
first pass, then optional LLM classification for borderline cases via
OpenRouter. Configurable sensitivity (strict/moderate/lenient), editable
keyword lists, blocked domain list, and admin stats display. Includes
30 unit tests covering domain blocking, title/body keywords, URL paths,
edge cases, and prompt structure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use $wpdb->prepare() with esc_like() in clear_article_cache()
- Add wp_http_validate_url() SSRF prevention in download_thumbnail()
- Add limit_response_size (5MB) on image downloads
- Set chmod 0644 after file_put_contents()
- Add transient-based concurrent fetch lock
