peptide-repo/prautoblogger
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update ARCHITECTURE.md for PullPush.io migration

Browse Source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
peptiderepo
2026-04-12 10:00:59 +07:00
parent 0db5096b31
commit fd404a26ab
1 changed files with 234 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files

131
ARCHITECTURE.md
View File

@@ -15,7 +15,8 @@ PRAutoBlogger is a WordPress plugin that monitors social media (starting with Re
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ 1. Source Collector │ Pulls raw posts/comments from social platforms │ 1. Source Collector │ Pulls raw posts/comments from social platforms
│ (Reddit API, etc.) │ Stores raw data in `ab_source_data` table │ (PullPush.io primary │ PullPush.io API (primary) / Reddit .json (fallback)
│ / Reddit .json fallback) Stores raw data in `ab_source_data` table
└────────────┬────────────┘ └────────────┬────────────┘
│ │
▼ ▼
@@ -127,8 +128,9 @@ prautoblogger/
│ │ ├── class-openrouter-provider.php # OpenRouter API implementation │ │ ├── class-openrouter-provider.php # OpenRouter API implementation
│ │ ├── class-openrouter-pricing.php # Model pricing lookup and cost estimation │ │ ├── class-openrouter-pricing.php # Model pricing lookup and cost estimation
│ │ ├── interface-source-provider.php # Contract for any social media source │ │ ├── interface-source-provider.php # Contract for any social media source
│ │ ├── class-reddit-provider.php # Reddit API (OAuth) implementation │ │ ├── class-pull-push-client.php # PullPush.io HTTP client (primary, no auth)
│ │ ├── class-reddit-api-client.php # Reddit HTTP client (OAuth + rate limiting) │ │ ├── class-reddit-json-client.php # Reddit .json endpoint client (fallback, no auth)
│ │ ├── class-reddit-provider.php # Reddit data collection (PullPush primary + .json fallback)
│ │ ├── class-tiktok-provider.php # TikTok — stub/future implementation │ │ ├── class-tiktok-provider.php # TikTok — stub/future implementation
│ │ ├── class-instagram-provider.php # Instagram — stub/future implementation │ │ ├── class-instagram-provider.php # Instagram — stub/future implementation
│ │ └── class-youtube-provider.php # YouTube — stub/future implementation │ │ └── class-youtube-provider.php # YouTube — stub/future implementation
@@ -191,20 +193,21 @@ prautoblogger/
└──────────────────────────────┘ └──────────────────────────────┘
``` ```
### Shortcode Attributes **Shortcode Attributes**
| Attribute | Default | Description | | Attribute | Default | Description |
|------------|---------|-------------| |------------|----------------------------------|-------------------------|
| `count` | 6 | Posts to display (max 12) | | `count` | `6` | Posts to display (max 12) |
| `category` | (all) | Filter by category slug | | `category` | (all) | Filter by category slug |
| `title` | "Latest Research & Insights" | Widget heading | | `title` | `"Latest Research & Insights"` | Widget heading |
| `subtitle` | "Evidence-based articles..." | Widget subheading | | `subtitle` | `"Evidence-based articles..."` | Widget subheading |
### CSS Integration **CSS Integration**
The widget CSS (`posts-widget.css`) uses the Peptide Starter theme's CSS custom properties (`--color-*`, `--spacing-*`, `--text-*`, `--radius-*`, `--transition-*`) with hardcoded fallbacks. This means: The widget CSS (`posts-widget.css`) uses the Peptide Starter theme's CSS custom properties (`--color-*`, `--spacing-*`, `--text-*`, `--radius-*`, `--transition-*`) with hardcoded fallbacks. This means:
- On sites using Peptide Starter: widget automatically adapts to light/dark mode via `data-theme`.
- On other themes: fallback values render a sensible dark-themed design. - **On sites using Peptide Starter:** widget automatically adapts to light/dark mode via `data-theme`.
- **On other themes:** fallback values render a sensible dark-themed design.
--- ---
@@ -215,8 +218,9 @@ The widget CSS (`posts-widget.css`) uses the Peptide Starter theme's CSS custom
All tables use `$wpdb->prefix` + `prautoblogger_` prefix. All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
#### `ab_source_data` — Raw social media data #### `ab_source_data` — Raw social media data
| Column | Type | Description | | Column | Type | Description |
|-----------------|-------------------|------------------------------------------------| |---------------|-------------------|--------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK | | id | BIGINT UNSIGNED | Auto-increment PK |
| source_type | VARCHAR(50) | 'reddit', 'tiktok', 'instagram', 'youtube' | | source_type | VARCHAR(50) | 'reddit', 'tiktok', 'instagram', 'youtube' |
| source_id | VARCHAR(255) | Platform-specific unique ID (post/comment ID) | | source_id | VARCHAR(255) | Platform-specific unique ID (post/comment ID) |
@@ -229,12 +233,14 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| permalink | VARCHAR(500) | URL to original content | | permalink | VARCHAR(500) | URL to original content |
| collected_at | DATETIME | When we collected this data | | collected_at | DATETIME | When we collected this data |
| metadata_json | LONGTEXT | Platform-specific extra data (JSON) | | metadata_json | LONGTEXT | Platform-specific extra data (JSON) |
| INDEX | source_type, collected_at |
| UNIQUE INDEX | source_type, source_id | - INDEX: `source_type`, `collected_at`
- UNIQUE INDEX: `source_type`, `source_id`
#### `ab_analysis_results` — Analyzed topics and patterns #### `ab_analysis_results` — Analyzed topics and patterns
| Column | Type | Description | | Column | Type | Description |
|-----------------|-------------------|------------------------------------------------| |------------------|-------------------|------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK | | id | BIGINT UNSIGNED | Auto-increment PK |
| analysis_type | VARCHAR(50) | 'question', 'complaint', 'comparison' | | analysis_type | VARCHAR(50) | 'question', 'complaint', 'comparison' |
| topic | VARCHAR(500) | Detected topic/theme | | topic | VARCHAR(500) | Detected topic/theme |
@@ -244,12 +250,14 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| source_ids_json | LONGTEXT | JSON array of ab_source_data.id references | | source_ids_json | LONGTEXT | JSON array of ab_source_data.id references |
| analyzed_at | DATETIME | When analysis was performed | | analyzed_at | DATETIME | When analysis was performed |
| metadata_json | LONGTEXT | Extra analysis data (JSON) | | metadata_json | LONGTEXT | Extra analysis data (JSON) |
| INDEX | analysis_type, relevance_score |
| INDEX | analyzed_at | - INDEX: `analysis_type`, `relevance_score`
- INDEX: `analyzed_at`
#### `prab_generation_log` — API cost and generation tracking #### `prab_generation_log` — API cost and generation tracking
| Column | Type | Description | | Column | Type | Description |
|------------------|-------------------|-----------------------------------------------| |-------------------|-------------------|-----------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK | | id | BIGINT UNSIGNED | Auto-increment PK |
| post_id | BIGINT UNSIGNED | WordPress post ID (nullable, set after publish)| | post_id | BIGINT UNSIGNED | WordPress post ID (nullable, set after publish)|
| run_id | VARCHAR(36) | UUID linking all entries from one pipeline run | | run_id | VARCHAR(36) | UUID linking all entries from one pipeline run |
@@ -263,26 +271,30 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| response_status | VARCHAR(20) | 'success', 'error', 'timeout' | | response_status | VARCHAR(20) | 'success', 'error', 'timeout' |
| error_message | TEXT | Error details if failed (nullable) | | error_message | TEXT | Error details if failed (nullable) |
| created_at | DATETIME | When the API call was made | | created_at | DATETIME | When the API call was made |
| INDEX | post_id |
| INDEX | run_id | - INDEX: `post_id`
| INDEX | created_at | - INDEX: `run_id`
| INDEX | stage | - INDEX: `created_at`
- INDEX: `stage`
#### `prab_event_log` — Structured application logging #### `prab_event_log` — Structured application logging
| Column | Type | Description | | Column | Type | Description |
|------------------|-------------------|-----------------------------------------------| |------------|-------------------|-------------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK | | id | BIGINT UNSIGNED | Auto-increment PK |
| level | VARCHAR(10) | 'error', 'warning', 'info', 'debug' | | level | VARCHAR(10) | 'error', 'warning', 'info', 'debug' |
| context | VARCHAR(100) | Origin: 'pipeline', 'reddit', 'ga4', etc. | | context | VARCHAR(100) | Origin: 'pipeline', 'reddit', 'ga4', etc. |
| message | TEXT | Human-readable log message | | message | TEXT | Human-readable log message |
| meta_json | LONGTEXT | Optional structured data (nullable) | | meta_json | LONGTEXT | Optional structured data (nullable) |
| created_at | DATETIME | When the event occurred | | created_at | DATETIME | When the event occurred |
| INDEX | level, created_at |
| INDEX | created_at | - INDEX: `level`, `created_at`
- INDEX: `created_at`
#### `ab_content_scores` — Post performance metrics #### `ab_content_scores` — Post performance metrics
| Column | Type | Description | | Column | Type | Description |
|------------------|-------------------|-----------------------------------------------| |--------------------|-------------------|----------------------------------------------|
| id | BIGINT UNSIGNED | Auto-increment PK | | id | BIGINT UNSIGNED | Auto-increment PK |
| post_id | BIGINT UNSIGNED | WordPress post ID | | post_id | BIGINT UNSIGNED | WordPress post ID |
| pageviews | INT | Total pageviews (WP native or GA4) | | pageviews | INT | Total pageviews (WP native or GA4) |
@@ -292,20 +304,18 @@ All tables use `$wpdb->prefix` + `prautoblogger_` prefix.
| composite_score | FLOAT | LLM-computed content quality score (0-100) | | composite_score | FLOAT | LLM-computed content quality score (0-100) |
| score_factors_json | LONGTEXT | JSON breakdown of what contributed to score | | score_factors_json | LONGTEXT | JSON breakdown of what contributed to score |
| measured_at | DATETIME | When metrics were collected | | measured_at | DATETIME | When metrics were collected |
| INDEX | post_id |
| INDEX | measured_at | - INDEX: `post_id`
| INDEX | composite_score | - INDEX: `measured_at`
- INDEX: `composite_score`
### WordPress Options (`wp_options`) ### WordPress Options (`wp_options`)
All prefixed with `prautoblogger_`: All prefixed with `prautoblogger_`:
| Option Key | Description | | Option Key | Description |
|-------------------------------------|------------------------------------------------------| |----------------------------------------|-------------------------------------------------------|
| `prautoblogger_openrouter_api_key` | Encrypted OpenRouter API key | | `prautoblogger_openrouter_api_key` | Encrypted OpenRouter API key |
| `prautoblogger_reddit_client_id` | Reddit OAuth client ID |
| `prautoblogger_reddit_client_secret` | Encrypted Reddit client secret |
| `prautoblogger_reddit_access_token` | Reddit OAuth access token (transient-backed) |
| `prautoblogger_ga4_property_id` | Google Analytics 4 property ID | | `prautoblogger_ga4_property_id` | Google Analytics 4 property ID |
| `prautoblogger_ga4_credentials_json` | Encrypted GA4 service account credentials | | `prautoblogger_ga4_credentials_json` | Encrypted GA4 service account credentials |
| `prautoblogger_analysis_model` | OpenRouter model for analysis (default: cheap) | | `prautoblogger_analysis_model` | OpenRouter model for analysis (default: cheap) |
@@ -333,7 +343,7 @@ All prefixed with `prautoblogger_`:
Stored on every PRAutoBlogger-generated post: Stored on every PRAutoBlogger-generated post:
| Meta Key | Description | | Meta Key | Description |
|---------------------------------------|----------------------------------------------| |---------------------------------------|-----------------------------------------------|
| `_prautoblogger_generated` | Boolean flag — '1' if generated by plugin | | `_prautoblogger_generated` | Boolean flag — '1' if generated by plugin |
| `_prautoblogger_analysis_id` | FK to ab_analysis_results.id | | `_prautoblogger_analysis_id` | FK to ab_analysis_results.id |
| `_prautoblogger_source_ids` | JSON array of source data IDs used | | `_prautoblogger_source_ids` | JSON array of source data IDs used |
@@ -350,38 +360,55 @@ Stored on every PRAutoBlogger-generated post:
## External API Integrations ## External API Integrations
| API | Purpose | Code Location | | Service | Purpose | Auth | Rate Limit | Code |
|----------------|-----------------------------------|----------------------------------------| |---------|---------|------|------------|------|
| OpenRouter | All LLM calls (analysis, writing, editing) | `providers/class-openrouter-provider.php`, `providers/class-openrouter-pricing.php` | | OpenRouter | All LLM calls (analysis, writing, editing) | API key (encrypted in wp_options) | Per-model | `providers/class-openrouter-provider.php`, `providers/class-openrouter-pricing.php` |
| Reddit API | Pull posts/comments from subreddits | `providers/class-reddit-provider.php`, `providers/class-reddit-api-client.php` | | PullPush.io | Primary Reddit data source — submissions and comments by subreddit | None (free public API) | 30 req/min, 1000/hour | `providers/class-pull-push-client.php` |
| Google Analytics 4 | Post performance metrics | `core/class-ga4-client.php`, `core/class-metrics-collector.php` | | Reddit .json | Fallback Reddit data — appends .json to Reddit URLs | None (unauthenticated) | ~10 req/min (datacenter IPs may be blocked) | `providers/class-reddit-json-client.php` |
| Google Analytics 4 | Post performance metrics | OAuth2 service account | Standard GA4 limits | `core/class-ga4-client.php`, `core/class-metrics-collector.php` |
--- ---
## Key Decisions ## Key Decisions
1. **OpenRouter over direct provider APIs.** Gives access to many models through one API, simplifies provider management, and lets the user switch models without code changes. Trade-off: slight latency overhead and dependency on OpenRouter's availability. ### #1: OpenRouter over direct provider APIs
Gives access to many models through one API, simplifies provider management, and lets the user switch models without code changes. Trade-off: slight latency overhead and dependency on OpenRouter's availability.
2. **Reddit OAuth API over JSON scraping.** Official API is reliable, rate-limited in a predictable way, and won't break unexpectedly. Requires user to create a Reddit app (we document this). Trade-off: more setup friction for the user. ### #2: Reddit as first social source (via PullPush.io + .json fallback)
Reddit has rich discussion data ideal for identifying recurring questions and pain points. PullPush.io is the primary data source (free, no auth required). Reddit .json endpoints serve as a fallback. See Key Decision #14 for the migration from Reddit OAuth. Trade-off: Reddit data is text-heavy (no video transcription needed), making it the simplest starting point.
3. **Chief editor agent instead of human review queue.** The user wants full automation — a second LLM pass reviews quality, SEO, and accuracy before publishing. Posts that fail editorial review are flagged for human intervention rather than published. Trade-off: doubles the LLM cost of the "review" step, but catches quality issues. ### #3: Chief editor agent instead of human review queue
The user wants full automation — a second LLM pass reviews quality, SEO, and accuracy before publishing. Posts that fail editorial review are flagged for human intervention rather than published. Trade-off: doubles the LLM cost of the "review" step, but catches quality issues.
4. **Configurable pipeline (single-pass vs. multi-step).** Some users want cheap/fast, others want high quality. Making this configurable avoids forcing one approach. Trade-off: more code paths to maintain. ### #4: Configurable pipeline (single-pass vs. multi-step)
Some users want cheap/fast, others want high quality. Making this configurable avoids forcing one approach. Trade-off: more code paths to maintain.
5. **Composite content score using LLM.** Rather than just tracking raw metrics, we periodically have the LLM evaluate what made high-performing posts succeed and low-performing posts fail. This feeds back into the analysis and generation prompts. Trade-off: additional API cost for scoring, but enables the self-improvement loop. ### #5: Composite content score using LLM
Rather than just tracking raw metrics, we periodically have the LLM evaluate what made high-performing posts succeed and low-performing posts fail. This feeds back into the analysis and generation prompts. Trade-off: additional API cost for scoring, but enables the self-improvement loop.
6. **Source provider interface for future platforms.** Reddit is first, but TikTok/Instagram/YouTube are planned. The interface pattern means adding a new source is one class implementation. Trade-off: slight over-engineering for day one, but pays off immediately at source #2. ### #6: Source provider interface for future platforms
Reddit is first, but TikTok/Instagram/YouTube are planned. The interface pattern means adding a new source is one class implementation. Trade-off: slight over-engineering for day one, but pays off immediately at source #2.
7. **Custom tables for high-volume data.** Source data, analysis results, and generation logs are high-write, time-series data. WordPress post_meta and options are wrong for this. Custom tables with proper indexes. Trade-off: more complex activation/uninstall, but correct. ### #7: Custom tables for high-volume data
Source data, analysis results, and generation logs are high-write, time-series data. WordPress post_meta and options are wrong for this. Custom tables with proper indexes. Trade-off: more complex activation/uninstall, but correct.
8. **All API keys encrypted at rest.** Using `wp_salt()` as encryption key with OpenSSL. Not bulletproof (salt is in wp-config.php) but significantly better than plaintext in wp_options. Trade-off: adds complexity to option get/set, but necessary for security. ### #8: All API keys encrypted at rest
Using `wp_salt()` as encryption key with OpenSSL. Not bulletproof (salt is in wp-config.php) but significantly better than plaintext in wp_options. Trade-off: adds complexity to option get/set, but necessary for security.
9. **Structured logger instead of raw error_log().** All application logging flows through `PRAutoBlogger_Logger` singleton, which writes to the `prab_event_log` table and forwards errors/warnings to PHP's `error_log()`. This gives users an in-admin Activity Log page with filtering by level, search, and pagination — much more accessible than server logs. Trade-off: one extra DB write per log entry, but the table is prunable and indexed. ### #9: Structured logger instead of raw error_log()
All application logging flows through `PRAutoBlogger_Logger` singleton, which writes to the `prab_event_log` table and forwards errors/warnings to PHP's `error_log()`. This gives users an in-admin Activity Log page with filtering by level, search, and pagination — much more accessible than server logs. Trade-off: one extra DB write per log entry, but the table is prunable and indexed.
10. **Database-level atomic mutex for generation lock.** Uses `INSERT IGNORE` on `wp_options` (which has a UNIQUE index on `option_name`) instead of transient-based locking. This eliminates the TOCTOU race condition where two concurrent cron runs could both read the transient as "not locked" before either sets it. Expired locks older than 1 hour are cleaned up first to prevent permanent deadlock. ### #10: Database-level atomic mutex for generation lock
Uses `INSERT IGNORE` on wp_options (which has a UNIQUE index on `option_name`) instead of transient-based locking. This eliminates the TOCTOU race condition where two concurrent cron runs could both read the transient as "not locked" before either sets it. Expired locks older than 1 hour are cleaned up first to prevent permanent deadlock.
12. **wp.element for frontend React (no build step).** The posts widget uses WordPress-bundled React (`wp.element`) with raw `createElement` calls instead of JSX. This avoids requiring a Node.js build pipeline for what is a simple card grid. Trade-off: more verbose component code, but zero tooling dependencies for the plugin consumer. ### #11: wp.element for frontend React (no build step)
The posts widget uses WordPress-bundled React (`wp.element`) with raw `createElement` calls instead of JSX. This avoids requiring a Node.js build pipeline for what is a simple card grid. Trade-off: more verbose component code, but zero tooling dependencies for the plugin consumer.
13. **REST API for frontend data fetching.** The widget fetches posts asynchronously from a dedicated REST endpoint rather than rendering server-side. This keeps initial page load fast and allows the endpoint to be cached independently (5-minute `Cache-Control`). The permission callback uses a filter (`prautoblogger_rest_posts_public`) so sites can restrict access if needed. ### #12: REST API for frontend data fetching
The widget fetches posts asynchronously from a dedicated REST endpoint rather than rendering server-side. This keeps initial page load fast and allows the endpoint to be cached independently (5-minute Cache-Control). The permission callback uses a filter (`prautoblogger_rest_posts_public`) so sites can restrict access if needed.
11. **Run-ID based log linking.** Each pipeline execution generates a UUID (`run_id`) that tags every `prab_generation_log` entry. When a post is published, `link_generation_logs()` uses `UPDATE WHERE run_id = X` to associate entries with the post_id. This is more reliable than the previous timestamp-window approach, especially in batch runs where multiple posts are generated in quick succession. ### #13: Run-ID based log linking
Each pipeline execution generates a UUID (`run_id`) that tags every `prab_generation_log` entry. When a post is published, `link_generation_logs()` uses `UPDATE WHERE run_id = X` to associate entries with the `post_id`. This is more reliable than the previous timestamp-window approach, especially in batch runs where multiple posts are generated in quick succession.
### #14: PullPush.io replaces Reddit OAuth
Reddit rejected our OAuth API application (April 2026). Rather than reapply or use a third-party proxy, we switched to PullPush.io as the primary data source. PullPush mirrors Reddit's submission and comment data via a free, no-auth API with 30 req/min limits. Reddit's .json endpoints (appending .json to any Reddit URL) serve as a fallback when PullPush is unavailable. The provider auto-detects PullPush availability on each collection run (cached for 5 minutes) and transparently falls back. Each collected item's metadata includes a `data_source` field (`pullpush` or `reddit_json`) for auditability. This eliminates the need for Reddit client ID/secret storage and the OAuth token lifecycle.