From fe8c242a648a2a713421b683371e6b85247012ca Mon Sep 17 00:00:00 2001 From: peptiderepo Date: Thu, 23 Apr 2026 14:51:51 +0000 Subject: [PATCH] Opik Wave 1 Commit 3: Version bump, docs, and hook registration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Version & Changelog: - prautoblogger.php: 0.11.0 → 0.12.0 - Add Opik PHP constants (PRAUTOBLOGGER_OPIK_API_KEY, WORKSPACE, URL_OVERRIDE) with fallback empty defaults so wp-config.php can override - CHANGELOG.md: v0.12.0 section describing Wave 1 scope, feature flag, observability benefits, cost model, and Wave 2 preview Documentation: - ARCHITECTURE.md: New 'Opik LLM Observability (Optional)' section with feature-flag design, credential isolation, span lifecycle, call sites, cost/performance implications, and code structure overview - CONVENTIONS.md: Step-by-step guide 'Instrumenting a New LLM Call Site' with code examples, key rules (no full prompts, async dispatch, separate cost tracking), and minimal instrumentation pattern Hook Registration: - class-prautoblogger.php: Register 'prautoblogger_opik_dispatch' cron action * Instantiates Opik_Client, Opik_Span_Queue, Opik_Dispatcher * Checks credentials and calls dispatcher->dispatch() * Updates 'prautoblogger_opik_last_dispatch' option for admin status display - Instantiate Opik_Settings in admin_init for admin UI registration All changes preserve ≤ 300-line files. Part of Wave 1 (observability skeleton). Target: v0.12.0 --- ARCHITECTURE.md | 44 +++++++++++++ CHANGELOG.md | 31 ++++++++++ CONVENTIONS.md | 102 ++++++++++++++++++++++++++++++- includes/class-prautoblogger.php | 24 ++++++++ prautoblogger.php | 19 +++++- 5 files changed, 217 insertions(+), 3 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 92ed361..b5e417f 100755 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -480,6 +480,50 @@ Stored on every PRAutoBlogger-generated post: --- +## Opik LLM Observability (Optional) + +**Status:** Wave 1 (v0.12.0) — tracing skeleton. Wave 2 (future) — eval harness with frozen-dataset regression testing. + +Opik is a cloud-hosted LLM observability platform (Comet) providing: +- **Per-call tracing:** Every OpenRouter call generates a span with model, tokens, latency metadata. +- **Trace grouping:** One trace per article generation bundles all spans (draft, QA, image prompt). +- **Prompt versioning:** Foundation for Wave 2 — compare new prompt revisions against a frozen reference dataset using LLM-as-judge scoring on rubric axes (factual accuracy, style match, hallucination risk). + +**Architecture:** +- **Feature flag:** `prautoblogger_opik_enabled` (WP option, default off). When false, zero network traffic to Opik. +- **Credentials:** API key and workspace from PHP constants `PRAUTOBLOGGER_OPIK_API_KEY`, `PRAUTOBLOGGER_OPIK_WORKSPACE` (defined in wp-config.php). Never stored in DB. +- **Span lifecycle:** + 1. `Opik_Trace_Context::init_trace()` generates UUID at article-worker start. + 2. Each LLM call wraps with `start_span()` → LLM request → `end_span()` with tokens + response. + 3. `finalize_trace()` + `Opik_Span_Queue::enqueue()` accumulates trace + spans for async dispatch. + 4. `prautoblogger_opik_dispatch` cron action batches spans/traces (max 100 per POST) to `/v1/private/spans/batch`, `/v1/private/traces/batch`. +- **Retry logic:** Exponential backoff (2s, 4s, 8s) on 5xx errors; permanent abort on 4xx; dropped after 3 attempts per item. +- **Queue:** WP options-backed; max 1000 items; 12-hour TTL; persists across page reloads for truly async dispatch. + +**Call sites instrumented (Wave 1):** +- Draft generation (single-pass or multi-step outline/draft/polish). +- Editorial review (chief editor LLM call). +- Image prompt generation (deferred; file at 314 LOC, over limit). + +**Cost & Performance:** +- Opik free tier: 25k spans/month. Estimated current usage ~4.2k/month (20 posts/day × 7 calls × 30 days). Headroom to 5× growth. +- Impact on article generation: async dispatch, zero blocking latency overhead. +- Wave 2 (eval harness): adds LLM-as-judge calls (~$14/month at Haiku pricing, assuming 2 eval runs/week). + +**Code structure:** +``` +includes/services/opik/ + class-opik-client.php — REST client (auth, retry, batch POSTs) + class-opik-trace-context.php — per-request trace state singleton + class-opik-span-queue.php — WP options-backed async queue + class-opik-dispatcher.php — cron handler, batch dispatch + +includes/admin/ + class-opik-settings.php — admin UI (toggle, project name, status) +``` + +--- + ## Key Decisions ### #1: OpenRouter over direct provider APIs diff --git a/CHANGELOG.md b/CHANGELOG.md index 0df268b..a93a4d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,37 @@ All notable changes to PRAutoBlogger will be documented in this file. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project uses [Semantic Versioning](https://semver.org/). +## [0.12.0] — 2026-04-23 + +### Added +- **Opik LLM observability integration (Wave 1: tracing skeleton).** Cloud-hosted observability via Comet Opik for detailed per-call tracing and prompt-version regression testing. Captured via async fire-and-forget dispatch to avoid blocking article generation. + +- **Opik REST client.** Minimal HTTP client (no third-party SDK) with exponential backoff retry, batch endpoints, and credential isolation (API key from wp-config.php constants, never in DB). + +- **Per-request trace context singleton.** Holds trace ID + span stack for one article generation. UUID threading across draft generation, editorial review, and image prompt generation. + +- **Async span/trace queue.** WP options-backed queue with TTL expiry (12 hours) and max depth (1000 items). Drained by cron dispatcher with per-batch retry logic. + +- **Cron dispatcher.** `prautoblogger_opik_dispatch` action scheduled via `wp_schedule_single_event()` drains queue in batches of up to 100 spans. + +- **Opik admin settings section.** Toggle to enable/disable tracing (default off), project name field, opt-in full-prompt capture, read-only status display (API key configured, last dispatch, queue depth). + +- **Span instrumentation at four LLM call sites:** single-pass article generation, outline/draft/polish stages (multi-step), and editorial review. Each span captures model name, provider, usage tokens. + +- **Comprehensive unit test suite.** 4 test files covering client (auth, retry, batch limits), trace context (UUID generation, span stacking), queue (enqueue/dequeue, expiry, reenqueue), and dispatcher (batch POST, separation of traces vs. spans). + +### Changed +- **class-article-worker.php:** Initializes Opik trace context at generation start, enqueues finalized trace + spans at end if feature enabled. + +- **class-content-generator.php:** Wraps single-pass and multi-step stages (outline, draft, polish) with Opik spans capturing token usage. + +- **class-chief-editor.php:** Wraps editorial review LLM call with Opik span. + +### Notes +- **Feature-flag default OFF.** Zero network traffic to Opik unless `prautoblogger_opik_enabled` option is true AND API credentials are defined in wp-config.php. +- **Image prompt generation instrumentation deferred.** File already at 314 LOC (over 300-line rule); can be instrumented in Wave 2 without modifying existing code. +- **Wave 2 holds eval harness.** CLI command for frozen-dataset regression testing with LLM-as-judge scoring (separate thread after v0.12.0 ships and is live-tested). + ## [0.11.0] — 2026-04-23 ### Added diff --git a/CONVENTIONS.md b/CONVENTIONS.md index 9cb417f..e260229 100644 --- a/CONVENTIONS.md +++ b/CONVENTIONS.md @@ -332,4 +332,104 @@ curl -s -X POST -H "Authorization: token $GH_TOKEN" \ ### When this changes -If the peptiderepo GitHub account is ever upgraded to Pro (or the repo goes public), replace this soft-enforcement section with a note pointing to the real branch protection rules in repo settings. \ No newline at end of file +If the peptiderepo GitHub account is ever upgraded to Pro (or the repo goes public), replace this soft-enforcement section with a note pointing to the real branch protection rules in repo settings. +## Instrumenting a New LLM Call Site for Opik (v0.12.0+) + +When you add a new LLM call (via `$llm->send_chat_completion()`), follow this pattern to capture it in Opik: + +### 1. Get the trace context + +At the start of your method, grab the per-request singleton: + +```php +$ctx = PRAutoBlogger_Opik_Trace_Context::current(); +``` + +### 2. Start the span + +Before the LLM call, declare a span with metadata: + +```php +$span_id = $ctx->start_span( + array( + 'name' => 'my_llm_operation', // kebab-case, descriptive + 'type' => 'llm', + 'model' => $model, // e.g. 'gpt-4', from get_option() + 'provider' => 'openrouter', // or the actual provider name + 'input' => array( 'key' => 'value' ), // optional metadata (no full prompts!) + ) +); +``` + +### 3. Make the LLM call (unchanged) + +```php +$response = $this->llm->send_chat_completion( + array( + array( 'role' => 'system', 'content' => $system_prompt ), + array( 'role' => 'user', 'content' => $user_prompt ), + ), + $model, + array( 'temperature' => 0.7, 'max_tokens' => 4000 ) +); +``` + +### 4. End the span with output + +After the response, record the outcome: + +```php +$ctx->end_span( + $span_id, + array( + 'output' => array( 'response_length' => strlen( $response['content'] ?? '' ) ), + 'usage' => array( + 'prompt_tokens' => $response['prompt_tokens'], + 'completion_tokens' => $response['completion_tokens'], + 'total_tokens' => $response['prompt_tokens'] + $response['completion_tokens'], + ), + ) +); +``` + +### Key rules + +- **Never log full prompts.** Use prompt hashes or version tags in `input`. Full text blows payload size and leaks content policy. +- **Check the feature flag at call-site level?** No. The singleton is always available; Opik_Dispatcher checks the flag before dispatch. +- **Async dispatch?** Yes. Spans are queued; dispatch happens via `prautoblogger_opik_dispatch` cron (no blocking). +- **Cost logging?** Unchanged. Your `cost_tracker->log_api_call()` remains separate from Opik; Opik is observability-only. + +### Example: minimal instrumentation + +```php +public function my_operation( $input ) { + $model = get_option( 'my_model', 'default-model' ); + $ctx = PRAutoBlogger_Opik_Trace_Context::current(); + + $span_id = $ctx->start_span( + array( + 'name' => 'my_operation', + 'type' => 'llm', + 'model' => $model, + 'provider' => 'openrouter', + ) + ); + + $response = $this->llm->send_chat_completion( /* ... */ ); + + $ctx->end_span( + $span_id, + array( + 'usage' => array( + 'prompt_tokens' => $response['prompt_tokens'], + 'completion_tokens' => $response['completion_tokens'], + 'total_tokens' => $response['prompt_tokens'] + $response['completion_tokens'], + ), + ) + ); + + return $response['content']; +} +``` + +That's it. No config, no feature-flag checks, no try/catch needed. diff --git a/includes/class-prautoblogger.php b/includes/class-prautoblogger.php index cdfc1b0..d021175 100644 --- a/includes/class-prautoblogger.php +++ b/includes/class-prautoblogger.php @@ -116,6 +116,30 @@ class PRAutoBlogger { $registry = $this->executor->get_model_registry(); add_action( 'prautoblogger_refresh_model_registry', array( $registry, 'refresh' ) ); + // Opik observability: async trace/span dispatch. + add_action( + 'prautoblogger_opik_dispatch', + static function (): void { + if ( ! defined( 'PRAUTOBLOGGER_OPIK_API_KEY' ) || empty( PRAUTOBLOGGER_OPIK_API_KEY ) ) { + return; + } + $client = new PRAutoBlogger_Opik_Client( + PRAUTOBLOGGER_OPIK_API_KEY, + PRAUTOBLOGGER_OPIK_WORKSPACE, + PRAUTOBLOGGER_OPIK_URL_OVERRIDE + ); + $queue = new PRAutoBlogger_Opik_Span_Queue(); + $dispatcher = new PRAutoBlogger_Opik_Dispatcher( $client, $queue ); + $dispatcher->dispatch(); + update_option( 'prautoblogger_opik_last_dispatch', time() ); + } + ); + + // Opik settings registration. + if ( is_admin() ) { + new PRAutoBlogger_Opik_Settings(); + } + // Single-idea generation from the Ideas browser page. add_action( 'prautoblogger_generate_from_idea', array( 'PRAutoBlogger_Ideas_Browser', 'on_cron_generate_from_idea' ) ); diff --git a/prautoblogger.php b/prautoblogger.php index 0c21ad6..83944a8 100644 --- a/prautoblogger.php +++ b/prautoblogger.php @@ -9,7 +9,7 @@ * Plugin Name: PRAutoBlogger * Plugin URI: https://peptiderepo.com/prautoblogger * Description: Monitors social media for trending topics, generates SEO-friendly blog posts using AI, and publishes them on a daily schedule with full cost tracking and self-improvement metrics. - * Version: 0.11.0 + * Version: 0.12.0 * Requires at least: 6.0 * Requires PHP: 7.4 * Author: PeptideRepo @@ -35,7 +35,7 @@ if ( ! defined( 'ABSPATH' ) ) { | Defined here so every file in the plugin can reference paths, versions, | and limits without magic strings. */ -define( 'PRAUTOBLOGGER_VERSION', '0.11.0' ); +define( 'PRAUTOBLOGGER_VERSION', '0.12.0' ); define( 'PRAUTOBLOGGER_DB_VERSION', '1.1.0' ); define( 'PRAUTOBLOGGER_PLUGIN_FILE', __FILE__ ); define( 'PRAUTOBLOGGER_PLUGIN_DIR', plugin_dir_path( __FILE__ ) ); @@ -66,6 +66,21 @@ define( 'PRAUTOBLOGGER_DEFAULT_IMAGE_STYLE_SUFFIX', 'Style: a single-panel newspaper comic in classic black ink on white paper, simple clean linework like The Far Side or B.C., slightly yellowed newsprint texture, hand-drawn crosshatching for shading, bold panel border, warm and humorous tone, visible in a newspaper comics section. IMPORTANT: Do not render any text, captions, speech bubbles, or words inside the image.' ); +// Opik observability (feature-flag gated). +// API credentials must be defined in wp-config.php constants: +// - PRAUTOBLOGGER_OPIK_API_KEY: API key for authentication +// - PRAUTOBLOGGER_OPIK_WORKSPACE: Opik workspace name (e.g. 'peptiderepo') +// - PRAUTOBLOGGER_OPIK_URL_OVERRIDE: Optional; defaults to https://www.comet.com/opik/api/ +if ( ! defined( 'PRAUTOBLOGGER_OPIK_API_KEY' ) ) { + define( 'PRAUTOBLOGGER_OPIK_API_KEY', '' ); +} +if ( ! defined( 'PRAUTOBLOGGER_OPIK_WORKSPACE' ) ) { + define( 'PRAUTOBLOGGER_OPIK_WORKSPACE', 'peptiderepo' ); +} +if ( ! defined( 'PRAUTOBLOGGER_OPIK_URL_OVERRIDE' ) ) { + define( 'PRAUTOBLOGGER_OPIK_URL_OVERRIDE', 'https://www.comet.com/opik/api/' ); +} + /* |-------------------------------------------------------------------------- | Autoloader