Fix thumbnail scraping: use search-based URL resolution for Google News articles
Google News RSS stores encrypted article URLs that cannot be resolved server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace the failed decode/parse strategies with DuckDuckGo search-based lookup using article title + publisher name to find actual article pages, then scrape OG images from those pages. - Add find_article_url_via_search() using DuckDuckGo HTML endpoint - Add resolve_article_for_scraping() combining redirect + search fallback - Mark failed lookups with '_no_image' sentinel to avoid repeated retries - Backfill button resets sentinels to allow manual re-scraping - Clean up verbose debug instrumentation from prior attempts - Update frontend to handle '_no_image' sentinel gracefully Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -95,7 +95,9 @@ class Peptide_News_Public {
|
||||
?>
|
||||
<div class="pn-news-feed pn-layout-<?php echo esc_attr( $layout ); ?>">
|
||||
<?php foreach ( $articles as $article ) :
|
||||
$thumb = ! empty( $article->thumbnail_url ) ? $article->thumbnail_url : $fallback_thumb;
|
||||
$thumb = ( ! empty( $article->thumbnail_url ) && '_no_image' !== $article->thumbnail_url )
|
||||
? $article->thumbnail_url
|
||||
: $fallback_thumb;
|
||||
$date = wp_date( 'M j, Y', strtotime( $article->published_at ) );
|
||||
?>
|
||||
<article class="pn-article" data-article-id="<?php echo esc_attr( $article->id ); ?>">
|
||||
|
||||
Reference in New Issue
Block a user