Terence Ong de35f2d684 Fix thumbnail scraping: use search-based URL resolution for Google News articles ...

Google News RSS stores encrypted article URLs that cannot be resolved
server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace
the failed decode/parse strategies with DuckDuckGo search-based lookup
using article title + publisher name to find actual article pages, then
scrape OG images from those pages.

- Add find_article_url_via_search() using DuckDuckGo HTML endpoint
- Add resolve_article_for_scraping() combining redirect + search fallback
- Mark failed lookups with '_no_image' sentinel to avoid repeated retries
- Backfill button resets sentinels to allow manual re-scraping
- Clean up verbose debug instrumentation from prior attempts
- Update frontend to handle '_no_image' sentinel gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-06 20:01:25 +07:00

13 KiB

Raw Blame History

View Raw