peptide-repo/prautoblogger
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update ARCHITECTURE.md for PullPush.io migration

Browse Source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
peptiderepo
2026-04-12 10:00:59 +07:00
parent 0db5096b31
commit fd404a26ab
1 changed files with 234 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files

141
ARCHITECTURE.md
View File

@@ -15,7 +15,8 @@ PRAutoBlogger is a WordPress plugin that monitors social media (starting with Re
▼
┌─────────────────────────┐
│ 1. Source Collector │ Pulls raw posts/comments from social platforms
│ (Reddit API, etc.) │ Stores raw data in `ab_source_data` table
│ (PullPush.io primary │ PullPush.io API (primary) / Reddit .json (fallback)
│ / Reddit .json fallback) Stores raw data in `ab_source_data` table
└────────────┬────────────┘
│
▼
@@ -113,22 +114,23 @@ prautoblogger/
│ │ ├── class-source-collector.php # Orchestrates data collection from all sources
│ │ ├── class-content-analyzer.php # LLM-powered analysis of collected social data
│ │ ├── class-idea-scorer.php # Ranks and deduplicates article ideas
│ │ ├── class-content-generator.php # Writer agent — manages the generation pipeline
│ │ ├── class-content-generator.php# Writer agent — manages the generation pipeline
│ │ ├── class-chief-editor.php # Editor agent — LLM-powered editorial review
│ │ ├── class-publisher.php # Creates WordPress posts from approved content
│ │ ├── class-cost-tracker.php # Logs all API costs, enforces budget limits
│ │ ├── class-logger.php # Structured logging singleton (error/warning/info/debug)
│ │ ├── class-pipeline-runner.php # Orchestrates the full generation pipeline
│ │ ├── class-ga4-client.php # Google Analytics 4 API client (OAuth + Data API)
│ │ └── class-metrics-collector.php # Collects post performance data (WP + GA4)
│ │ └── class-metrics-collector.php# Collects post performance data (WP + GA4)
│ │
│ ├── providers/
│ │ ├── interface-llm-provider.php # Contract for any LLM provider
│ │ ├── class-openrouter-provider.php # OpenRouter API implementation
│ │ ├── class-openrouter-pricing.php # Model pricing lookup and cost estimation
│ │ ├── interface-source-provider.php # Contract for any social media source
│ │ ├── class-reddit-provider.php # Reddit API (OAuth) implementation
│ │ ├── class-reddit-api-client.php # Reddit HTTP client (OAuth + rate limiting)
│ │ ├── class-pull-push-client.php # PullPush.io HTTP client (primary, no auth)
│ │ ├── class-reddit-json-client.php # Reddit .json endpoint client (fallback, no auth)
│ │ ├── class-reddit-provider.php # Reddit data collection (PullPush primary + .json fallback)
│ │ ├── class-tiktok-provider.php # TikTok — stub/future implementation
│ │ ├── class-instagram-provider.php # Instagram — stub/future implementation
│ │ └── class-youtube-provider.php # YouTube — stub/future implementation
@@ -151,7 +153,7 @@ prautoblogger/
│ ├── metrics-page.php # Metrics/cost dashboard template
│ ├── review-queue.php # Review queue table template
│ ├── log-viewer.php # Activity log viewer template
│ └── metabox-generation-info.php # Post metabox template
│ └── metabox-generation-info.php# Post metabox template
│
├── languages/
│ └── prautoblogger.pot # i18n translation template
@@ -191,20 +193,21 @@ prautoblogger/
└──────────────────────────────┘
```
### Shortcode Attributes
**Shortcode Attributes**
| Attribute | Default | Description |
|------------|---------|-------------|
| `count` | 6 | Posts to display (max 12) |
|------------|----------------------------------|-------------------------|
| `count` | `6` | Posts to display (max 12) |
| `category` | (all) | Filter by category slug |
| `title` | "Latest Research & Insights" | Widget heading |
| `subtitle` | "Evidence-based articles..." | Widget subheading |
| `title` | `"Latest Research & Insights"` | Widget heading |
| `subtitle` | `"Evidence-based articles..."` | Widget subheading |
### CSS Integration
**CSS Integration**
The widget CSS (`posts-widget.css`) uses the Peptide Starter theme's CSS custom properties (`--color-*`, `--spacing-*`, `--text-*`, `--radius-*`, `--transition-*`) with hardcoded fallbacks. This means:
- On sites using Peptide Starter: widget automatically adapts to light/dark mode via `data-theme`.
- On other themes: fallback values render a sensible dark-themed design.
- **On sites using Peptide Starter:** widget automatically adapts to light/dark mode via `data-theme`.
- **On other themes:** fallback values render a sensible dark-themed design.
---
@@ -215,8 +218,9 @@ The widget CSS (`posts-widget.css`) uses the Peptide Starter theme's CSS custom
All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
#### `ab_source_data` — Raw social media data
| Column | Type | Description |
|-----------------|-------------------|------------------------------------------------|
|---------------|-------------------|--------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK |
| source_type | VARCHAR(50) | 'reddit', 'tiktok', 'instagram', 'youtube' |
| source_id | VARCHAR(255) | Platform-specific unique ID (post/comment ID) |
@@ -229,12 +233,14 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| permalink | VARCHAR(500) | URL to original content |
| collected_at | DATETIME | When we collected this data |
| metadata_json | LONGTEXT | Platform-specific extra data (JSON) |
| INDEX | source_type, collected_at |
| UNIQUE INDEX | source_type, source_id |
- INDEX: `source_type`, `collected_at`
- UNIQUE INDEX: `source_type`, `source_id`
#### `ab_analysis_results` — Analyzed topics and patterns
| Column | Type | Description |
|-----------------|-------------------|------------------------------------------------|
|------------------|-------------------|------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK |
| analysis_type | VARCHAR(50) | 'question', 'complaint', 'comparison' |
| topic | VARCHAR(500) | Detected topic/theme |
@@ -244,12 +250,14 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| source_ids_json | LONGTEXT | JSON array of ab_source_data.id references |
| analyzed_at | DATETIME | When analysis was performed |
| metadata_json | LONGTEXT | Extra analysis data (JSON) |
| INDEX | analysis_type, relevance_score |
| INDEX | analyzed_at |
- INDEX: `analysis_type`, `relevance_score`
- INDEX: `analyzed_at`
#### `prab_generation_log` — API cost and generation tracking
| Column | Type | Description |
|------------------|-------------------|-----------------------------------------------|
|-------------------|-------------------|-----------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK |
| post_id | BIGINT UNSIGNED | WordPress post ID (nullable, set after publish)|
| run_id | VARCHAR(36) | UUID linking all entries from one pipeline run |
@@ -257,32 +265,36 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| provider | VARCHAR(50) | 'openrouter' |
| model | VARCHAR(100) | Model identifier used |
| prompt_tokens | INT | Input tokens consumed |
| completion_tokens| INT | Output tokens generated |
| completion_tokens | INT | Output tokens generated |
| estimated_cost | DECIMAL(10,6) | Estimated USD cost |
| request_json | LONGTEXT | Full request payload (for debugging) |
| response_status | VARCHAR(20) | 'success', 'error', 'timeout' |
| error_message | TEXT | Error details if failed (nullable) |
| created_at | DATETIME | When the API call was made |
| INDEX | post_id |
| INDEX | run_id |
| INDEX | created_at |
| INDEX | stage |
- INDEX: `post_id`
- INDEX: `run_id`
- INDEX: `created_at`
- INDEX: `stage`
#### `prab_event_log` — Structured application logging
| Column | Type | Description |
|------------------|-------------------|-----------------------------------------------|
|------------|-------------------|-------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK |
| level | VARCHAR(10) | 'error', 'warning', 'info', 'debug' |
| context | VARCHAR(100) | Origin: 'pipeline', 'reddit', 'ga4', etc. |
| message | TEXT | Human-readable log message |
| meta_json | LONGTEXT | Optional structured data (nullable) |
| created_at | DATETIME | When the event occurred |
| INDEX | level, created_at |
| INDEX | created_at |
- INDEX: `level`, `created_at`
- INDEX: `created_at`
#### `ab_content_scores` — Post performance metrics
| Column | Type | Description |
|------------------|-------------------|-----------------------------------------------|
|--------------------|-------------------|----------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK |
| post_id | BIGINT UNSIGNED | WordPress post ID |
| pageviews | INT | Total pageviews (WP native or GA4) |
@@ -290,22 +302,20 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| bounce_rate | FLOAT | Bounce rate percentage (GA4) |
| comment_count | INT | WordPress comment count |
| composite_score | FLOAT | LLM-computed content quality score (0-100) |
| score_factors_json| LONGTEXT | JSON breakdown of what contributed to score |
| score_factors_json | LONGTEXT | JSON breakdown of what contributed to score |
| measured_at | DATETIME | When metrics were collected |
| INDEX | post_id |
| INDEX | measured_at |
| INDEX | composite_score |
- INDEX: `post_id`
- INDEX: `measured_at`
- INDEX: `composite_score`
### WordPress Options (`wp_options`)
All prefixed with `prautoblogger_`:
| Option Key | Description |
|-------------------------------------|------------------------------------------------------|
|----------------------------------------|-------------------------------------------------------|
| `prautoblogger_openrouter_api_key` | Encrypted OpenRouter API key |
| `prautoblogger_reddit_client_id` | Reddit OAuth client ID |
| `prautoblogger_reddit_client_secret` | Encrypted Reddit client secret |
| `prautoblogger_reddit_access_token` | Reddit OAuth access token (transient-backed) |
| `prautoblogger_ga4_property_id` | Google Analytics 4 property ID |
| `prautoblogger_ga4_credentials_json` | Encrypted GA4 service account credentials |
| `prautoblogger_analysis_model` | OpenRouter model for analysis (default: cheap) |
@@ -333,7 +343,7 @@ All prefixed with `prautoblogger_`:
Stored on every PRAutoBlogger-generated post:
| Meta Key | Description |
|---------------------------------------|----------------------------------------------|
|---------------------------------------|-----------------------------------------------|
| `_prautoblogger_generated` | Boolean flag — '1' if generated by plugin |
| `_prautoblogger_analysis_id` | FK to ab_analysis_results.id |
| `_prautoblogger_source_ids` | JSON array of source data IDs used |
@@ -350,38 +360,55 @@ Stored on every PRAutoBlogger-generated post:
## External API Integrations
| API | Purpose | Code Location |
|----------------|-----------------------------------|----------------------------------------|
| OpenRouter | All LLM calls (analysis, writing, editing) | `providers/class-openrouter-provider.php`, `providers/class-openrouter-pricing.php` |
| Reddit API | Pull posts/comments from subreddits | `providers/class-reddit-provider.php`, `providers/class-reddit-api-client.php` |
| Google Analytics 4 | Post performance metrics | `core/class-ga4-client.php`, `core/class-metrics-collector.php` |
| Service | Purpose | Auth | Rate Limit | Code |
|---------|---------|------|------------|------|
| OpenRouter | All LLM calls (analysis, writing, editing) | API key (encrypted in wp_options) | Per-model | `providers/class-openrouter-provider.php`, `providers/class-openrouter-pricing.php` |
| PullPush.io | Primary Reddit data source — submissions and comments by subreddit | None (free public API) | 30 req/min, 1000/hour | `providers/class-pull-push-client.php` |
| Reddit .json | Fallback Reddit data — appends .json to Reddit URLs | None (unauthenticated) | ~10 req/min (datacenter IPs may be blocked) | `providers/class-reddit-json-client.php` |
| Google Analytics 4 | Post performance metrics | OAuth2 service account | Standard GA4 limits | `core/class-ga4-client.php`, `core/class-metrics-collector.php` |
---
## Key Decisions
1. **OpenRouter over direct provider APIs.** Gives access to many models through one API, simplifies provider management, and lets the user switch models without code changes. Trade-off: slight latency overhead and dependency on OpenRouter's availability.
### #1: OpenRouter over direct provider APIs
Gives access to many models through one API, simplifies provider management, and lets the user switch models without code changes. Trade-off: slight latency overhead and dependency on OpenRouter's availability.
2. **Reddit OAuth API over JSON scraping.** Official API is reliable, rate-limited in a predictable way, and won't break unexpectedly. Requires user to create a Reddit app (we document this). Trade-off: more setup friction for the user.
### #2: Reddit as first social source (via PullPush.io + .json fallback)
Reddit has rich discussion data ideal for identifying recurring questions and pain points. PullPush.io is the primary data source (free, no auth required). Reddit .json endpoints serve as a fallback. See Key Decision #14 for the migration from Reddit OAuth. Trade-off: Reddit data is text-heavy (no video transcription needed), making it the simplest starting point.
3. **Chief editor agent instead of human review queue.** The user wants full automation — a second LLM pass reviews quality, SEO, and accuracy before publishing. Posts that fail editorial review are flagged for human intervention rather than published. Trade-off: doubles the LLM cost of the "review" step, but catches quality issues.
### #3: Chief editor agent instead of human review queue
The user wants full automation — a second LLM pass reviews quality, SEO, and accuracy before publishing. Posts that fail editorial review are flagged for human intervention rather than published. Trade-off: doubles the LLM cost of the "review" step, but catches quality issues.
4. **Configurable pipeline (single-pass vs. multi-step).** Some users want cheap/fast, others want high quality. Making this configurable avoids forcing one approach. Trade-off: more code paths to maintain.
### #4: Configurable pipeline (single-pass vs. multi-step)
Some users want cheap/fast, others want high quality. Making this configurable avoids forcing one approach. Trade-off: more code paths to maintain.
5. **Composite content score using LLM.** Rather than just tracking raw metrics, we periodically have the LLM evaluate what made high-performing posts succeed and low-performing posts fail. This feeds back into the analysis and generation prompts. Trade-off: additional API cost for scoring, but enables the self-improvement loop.
### #5: Composite content score using LLM
Rather than just tracking raw metrics, we periodically have the LLM evaluate what made high-performing posts succeed and low-performing posts fail. This feeds back into the analysis and generation prompts. Trade-off: additional API cost for scoring, but enables the self-improvement loop.
6. **Source provider interface for future platforms.** Reddit is first, but TikTok/Instagram/YouTube are planned. The interface pattern means adding a new source is one class implementation. Trade-off: slight over-engineering for day one, but pays off immediately at source #2.
### #6: Source provider interface for future platforms
Reddit is first, but TikTok/Instagram/YouTube are planned. The interface pattern means adding a new source is one class implementation. Trade-off: slight over-engineering for day one, but pays off immediately at source #2.
7. **Custom tables for high-volume data.** Source data, analysis results, and generation logs are high-write, time-series data. WordPress post_meta and options are wrong for this. Custom tables with proper indexes. Trade-off: more complex activation/uninstall, but correct.
### #7: Custom tables for high-volume data
Source data, analysis results, and generation logs are high-write, time-series data. WordPress post_meta and options are wrong for this. Custom tables with proper indexes. Trade-off: more complex activation/uninstall, but correct.
8. **All API keys encrypted at rest.** Using `wp_salt()` as encryption key with OpenSSL. Not bulletproof (salt is in wp-config.php) but significantly better than plaintext in wp_options. Trade-off: adds complexity to option get/set, but necessary for security.
### #8: All API keys encrypted at rest
Using `wp_salt()` as encryption key with OpenSSL. Not bulletproof (salt is in wp-config.php) but significantly better than plaintext in wp_options. Trade-off: adds complexity to option get/set, but necessary for security.
9. **Structured logger instead of raw error_log().** All application logging flows through `PRAutoBlogger_Logger` singleton, which writes to the `prab_event_log` table and forwards errors/warnings to PHP's `error_log()`. This gives users an in-admin Activity Log page with filtering by level, search, and pagination — much more accessible than server logs. Trade-off: one extra DB write per log entry, but the table is prunable and indexed.
### #9: Structured logger instead of raw error_log()
All application logging flows through `PRAutoBlogger_Logger` singleton, which writes to the `prab_event_log` table and forwards errors/warnings to PHP's `error_log()`. This gives users an in-admin Activity Log page with filtering by level, search, and pagination — much more accessible than server logs. Trade-off: one extra DB write per log entry, but the table is prunable and indexed.
10. **Database-level atomic mutex for generation lock.** Uses `INSERT IGNORE` on `wp_options` (which has a UNIQUE index on `option_name`) instead of transient-based locking. This eliminates the TOCTOU race condition where two concurrent cron runs could both read the transient as "not locked" before either sets it. Expired locks older than 1 hour are cleaned up first to prevent permanent deadlock.
### #10: Database-level atomic mutex for generation lock
Uses `INSERT IGNORE` on wp_options (which has a UNIQUE index on `option_name`) instead of transient-based locking. This eliminates the TOCTOU race condition where two concurrent cron runs could both read the transient as "not locked" before either sets it. Expired locks older than 1 hour are cleaned up first to prevent permanent deadlock.
12. **wp.element for frontend React (no build step).** The posts widget uses WordPress-bundled React (`wp.element`) with raw `createElement` calls instead of JSX. This avoids requiring a Node.js build pipeline for what is a simple card grid. Trade-off: more verbose component code, but zero tooling dependencies for the plugin consumer.
### #11: wp.element for frontend React (no build step)
The posts widget uses WordPress-bundled React (`wp.element`) with raw `createElement` calls instead of JSX. This avoids requiring a Node.js build pipeline for what is a simple card grid. Trade-off: more verbose component code, but zero tooling dependencies for the plugin consumer.
13. **REST API for frontend data fetching.** The widget fetches posts asynchronously from a dedicated REST endpoint rather than rendering server-side. This keeps initial page load fast and allows the endpoint to be cached independently (5-minute `Cache-Control`). The permission callback uses a filter (`prautoblogger_rest_posts_public`) so sites can restrict access if needed.
### #12: REST API for frontend data fetching
The widget fetches posts asynchronously from a dedicated REST endpoint rather than rendering server-side. This keeps initial page load fast and allows the endpoint to be cached independently (5-minute Cache-Control). The permission callback uses a filter (`prautoblogger_rest_posts_public`) so sites can restrict access if needed.
11. **Run-ID based log linking.** Each pipeline execution generates a UUID (`run_id`) that tags every `prab_generation_log` entry. When a post is published, `link_generation_logs()` uses `UPDATE WHERE run_id = X` to associate entries with the post_id. This is more reliable than the previous timestamp-window approach, especially in batch runs where multiple posts are generated in quick succession.
### #13: Run-ID based log linking
Each pipeline execution generates a UUID (`run_id`) that tags every `prab_generation_log` entry. When a post is published, `link_generation_logs()` uses `UPDATE WHERE run_id = X` to associate entries with the `post_id`. This is more reliable than the previous timestamp-window approach, especially in batch runs where multiple posts are generated in quick succession.
### #14: PullPush.io replaces Reddit OAuth
Reddit rejected our OAuth API application (April 2026). Rather than reapply or use a third-party proxy, we switched to PullPush.io as the primary data source. PullPush mirrors Reddit's submission and comment data via a free, no-auth API with 30 req/min limits. Reddit's .json endpoints (appending .json to any Reddit URL) serve as a fallback when PullPush is unavailable. The provider auto-detects PullPush availability on each collection run (cached for 5 minutes) and transparently falls back. Each collected item's metadata includes a `data_source` field (`pullpush` or `reddit_json`) for auditability. This eliminates the need for Reddit client ID/secret storage and the OAuth token lifecycle.