Fix thumbnail scraping: use search-based URL resolution for Google News articles

Google News RSS stores encrypted article URLs that cannot be resolved
server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace
the failed decode/parse strategies with DuckDuckGo search-based lookup
using article title + publisher name to find actual article pages, then
scrape OG images from those pages.

- Add find_article_url_via_search() using DuckDuckGo HTML endpoint
- Add resolve_article_for_scraping() combining redirect + search fallback
- Mark failed lookups with '_no_image' sentinel to avoid repeated retries
- Backfill button resets sentinels to allow manual re-scraping
- Clean up verbose debug instrumentation from prior attempts
- Update frontend to handle '_no_image' sentinel gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Terence Ong
2026-04-06 20:01:25 +07:00
parent acf7e61b90
commit de35f2d684
3 changed files with 211 additions and 305 deletions

View File

@@ -95,7 +95,9 @@ class Peptide_News_Public {
?>
<div class="pn-news-feed pn-layout-<?php echo esc_attr( $layout ); ?>">
<?php foreach ( $articles as $article ) :
$thumb = ! empty( $article->thumbnail_url ) ? $article->thumbnail_url : $fallback_thumb;
$thumb = ( ! empty( $article->thumbnail_url ) && '_no_image' !== $article->thumbnail_url )
? $article->thumbnail_url
: $fallback_thumb;
$date = wp_date( 'M j, Y', strtotime( $article->published_at ) );
?>
<article class="pn-article" data-article-id="<?php echo esc_attr( $article->id ); ?>">