peptiderepo 281c8cd89c feat: semantic dedup + LLM-aware topic avoidance ...

Two-pronged fix for repeated article topics:

1. Feed last 30 days of published article titles into the LLM analysis
   prompt with explicit "avoid these" instruction. Zero cost — just
   extra context in an existing API call.

2. Replace keyword-overlap dedup (60% word match) with embedding-based
   cosine similarity via sentence-transformers/all-MiniLM-L12-v2 on
   OpenRouter. Catches rephrasings that keyword matching misses
   (e.g. "BPC-157 dosing" vs "how much BPC-157 to take"). Falls back
   to keyword matching if the embedding API is unavailable.

   Cost: ~$0.00001 per generation run ($0.02/M tokens, ~500 tokens/run).

New files:
- class-semantic-dedup.php — Embedding cosine similarity + keyword fallback
- class-open-router-embedding-provider.php — OpenRouter /embeddings client

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-19 13:35:26 +07:00

7.0 KiB

Executable File

Raw Blame History

View Raw