feat: semantic dedup + LLM-aware topic avoidance
Two-pronged fix for repeated article topics: 1. Feed last 30 days of published article titles into the LLM analysis prompt with explicit "avoid these" instruction. Zero cost — just extra context in an existing API call. 2. Replace keyword-overlap dedup (60% word match) with embedding-based cosine similarity via sentence-transformers/all-MiniLM-L12-v2 on OpenRouter. Catches rephrasings that keyword matching misses (e.g. "BPC-157 dosing" vs "how much BPC-157 to take"). Falls back to keyword matching if the embedding API is unavailable. Cost: ~$0.00001 per generation run ($0.02/M tokens, ~500 tokens/run). New files: - class-semantic-dedup.php — Embedding cosine similarity + keyword fallback - class-open-router-embedding-provider.php — OpenRouter /embeddings client Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -128,7 +128,8 @@ prautoblogger/
|
||||
│ │ ├── class-source-collector.php # Orchestrates data collection from all sources
|
||||
│ │ ├── class-content-analyzer.php # LLM-powered analysis of collected social data
|
||||
│ │ ├── class-analysis-prompts.php # System/user prompt builders + performance context for analyzer
|
||||
│ │ ├── class-idea-scorer.php # Ranks and deduplicates article ideas
|
||||
│ │ ├── class-idea-scorer.php # Ranks and deduplicates article ideas (delegates to Semantic_Dedup)
|
||||
│ │ ├── class-semantic-dedup.php # Embedding-based cosine similarity dedup with keyword fallback
|
||||
│ │ ├── class-content-generator.php# Writer agent — manages the generation pipeline
|
||||
│ │ ├── class-chief-editor.php # Editor agent — LLM-powered editorial review
|
||||
│ │ ├── class-publisher.php # Creates WordPress posts from approved content
|
||||
@@ -145,7 +146,8 @@ prautoblogger/
|
||||
│ │
|
||||
│ ├── providers/
|
||||
│ │ ├── interface-llm-provider.php # Contract for any LLM provider
|
||||
│ │ ├── class-open-router-provider.php # OpenRouter API implementation
|
||||
│ │ ├── class-open-router-provider.php # OpenRouter API implementation (chat completions)
|
||||
│ │ ├── class-open-router-embedding-provider.php # Text embeddings for semantic dedup
|
||||
│ │ ├── class-open-router-pricing.php # Model pricing lookup and cost estimation
|
||||
│ │ ├── interface-source-provider.php # Contract for any social media source
|
||||
│ │ ├── class-reddit-json-client.php # Reddit HTTP client — RSS (primary) + .json (fallback)
|
||||
|
||||
Reference in New Issue
Block a user