feat: semantic dedup + LLM-aware topic avoidance

Two-pronged fix for repeated article topics:

1. Feed last 30 days of published article titles into the LLM analysis
   prompt with explicit "avoid these" instruction. Zero cost — just
   extra context in an existing API call.

2. Replace keyword-overlap dedup (60% word match) with embedding-based
   cosine similarity via sentence-transformers/all-MiniLM-L12-v2 on
   OpenRouter. Catches rephrasings that keyword matching misses
   (e.g. "BPC-157 dosing" vs "how much BPC-157 to take"). Falls back
   to keyword matching if the embedding API is unavailable.

   Cost: ~$0.00001 per generation run ($0.02/M tokens, ~500 tokens/run).

New files:
- class-semantic-dedup.php — Embedding cosine similarity + keyword fallback
- class-open-router-embedding-provider.php — OpenRouter /embeddings client

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
peptiderepo
2026-04-19 13:35:26 +07:00
parent 00cfb39253
commit 281c8cd89c
5 changed files with 530 additions and 113 deletions

View File

@@ -128,7 +128,8 @@ prautoblogger/
│ │ ├── class-source-collector.php # Orchestrates data collection from all sources
│ │ ├── class-content-analyzer.php # LLM-powered analysis of collected social data
│ │ ├── class-analysis-prompts.php # System/user prompt builders + performance context for analyzer
│ │ ├── class-idea-scorer.php # Ranks and deduplicates article ideas
│ │ ├── class-idea-scorer.php # Ranks and deduplicates article ideas (delegates to Semantic_Dedup)
│ │ ├── class-semantic-dedup.php # Embedding-based cosine similarity dedup with keyword fallback
│ │ ├── class-content-generator.php# Writer agent — manages the generation pipeline
│ │ ├── class-chief-editor.php # Editor agent — LLM-powered editorial review
│ │ ├── class-publisher.php # Creates WordPress posts from approved content
@@ -145,7 +146,8 @@ prautoblogger/
│ │
│ ├── providers/
│ │ ├── interface-llm-provider.php # Contract for any LLM provider
│ │ ├── class-open-router-provider.php # OpenRouter API implementation
│ │ ├── class-open-router-provider.php # OpenRouter API implementation (chat completions)
│ │ ├── class-open-router-embedding-provider.php # Text embeddings for semantic dedup
│ │ ├── class-open-router-pricing.php # Model pricing lookup and cost estimation
│ │ ├── interface-source-provider.php # Contract for any social media source
│ │ ├── class-reddit-json-client.php # Reddit HTTP client — RSS (primary) + .json (fallback)