peptide-repo/peptide-news-plugin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix thumbnail scraping: use search-based URL resolution for Google News articles

Browse Source 
Google News RSS stores encrypted article URLs that cannot be resolved
server-side (JS SPA, no HTTP redirects, encrypted base64 IDs). Replace
the failed decode/parse strategies with DuckDuckGo search-based lookup
using article title + publisher name to find actual article pages, then
scrape OG images from those pages.

- Add find_article_url_via_search() using DuckDuckGo HTML endpoint
- Add resolve_article_for_scraping() combining redirect + search fallback
- Mark failed lookups with '_no_image' sentinel to avoid repeated retries
- Backfill button resets sentinels to allow manual re-scraping
- Clean up verbose debug instrumentation from prior attempts
- Update frontend to handle '_no_image' sentinel gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Terence Ong
2026-04-06 20:01:25 +07:00
parent acf7e61b90
commit de35f2d684
3 changed files with 211 additions and 305 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
admin/class-peptide-news-admin.php
View File

@@ -609,12 +609,10 @@ class Peptide_News_Admin {
$btn.prop('disabled', false).text('<?php echo esc_js( __( 'Fix Article Thumbnails', 'peptide-news' ) ); ?>');
if (response.success) {
var d = response.data;
var msg = d.updated + ' thumbnail(s) scraped. Query matched: ' + d.query_count;
if (d.total_active !== undefined) {
msg += ' | Total: ' + d.total_active + ', Has URL: ' + d.with_thumb_url + ', Has local: ' + d.with_thumb_local;
}
var msg = d.updated + ' of ' + d.total + ' thumbnails found.';
if (d.failed > 0) { msg += ' ' + d.failed + ' could not be resolved.'; }
$result.text(msg);
if (d.samples && d.samples.length) { console.log('Thumbnail debug:', JSON.stringify(d, null, 2)); }
if (d.samples && d.samples.length) { console.log('Thumbnail backfill:', JSON.stringify(d.samples, null, 2)); }
} else {
$result.text('Error: ' + (response.data || 'Unknown error'));
}