infra/vps: PRAB generation orchestrator (systemd timer, SSH-driven --sync) #27

2026-06-22T16:35:43Z

peptiderepo commented

2026-06-22 16:35:43 +00:00

(Migrated from github.com)

What changed

Adds infra/vps/ — the VPS-side orchestrator for Option A (ADR 2026-06-22): the Coolify KVM8 VPS (76.13.220.15) drives PRAutoBlogger article generation on prod synchronously over a held-open SSH session, escaping Hostinger's ~10-min background-PHP-process kill.

Files added

infra/vps/orchestrate-generation.sh

Invokes PRAutoBlogger_Run_Reaper::reap() via wp eval at run start (belt-and-suspenders reaper sweep).
SSH to Hostinger and runs wp prautoblogger generate --sync --path=... (requires companion plugin PR v0.22.1).
Lock-aware: plugin's Generation_Lock DB mutex prevents double-drive; --sync exits 0 with skipped JSON if a live run is in progress.
Structured log lines to stdout (captured by StandardOutput=journal → journalctl -u prab-generation).
Failure alert: sends email to cto@peptiderepo.com via mail (forwards to peptiderepo@gmail.com via Cloudflare Email Routing). No external API token needed; system mail is the alert channel.
DRY_RUN=1 mode: echoes SSH/WP-CLI commands without executing generation.
All config via env vars (SSH_KEY, SSH_HOST, SSH_PORT, SSH_USER, WP_PATH); no hardcoded secrets.
SSH key referenced by PATH (/root/.ssh/hostinger_prod_ed25519) — never embedded.
shellcheck-clean (SC2029 disable on the SSH command with remote variable expansion).

infra/vps/install-orchestrator.sh

One-time setup: copies orchestrator to /opt/prab-orchestrator/, installs .service + .timer to /etc/systemd/system/, runs daemon-reload.
Does NOT enable or start the timer — explicit post-QA cutover step.
Prints the exact cutover sequence for the operator.

infra/vps/prab-generation.service

Type=oneshot, User=root, TimeoutStartSec=7200 (covers 32-min LLM calls observed in prod).
Restart=no — timer retries next day on failure.
SyslogIdentifier=prab-generation for clean journal queries.

infra/vps/prab-generation.timer

OnCalendar=*-*-* 06:00:00 UTC (fires before Hostinger daily wp-cron window).
Persistent=true — if VPS was down at 06:00 UTC, fires immediately on next boot.
RandomizedDelaySec=300 — jitter for future multi-VPS scenarios.

Why

Hostinger's shared-hosting plan kills detached background PHP processes at ~10 min (CloudLinux LVE). A single LLM draft call observed 1925s (~32 min). SSH-attached processes survive because they run under a different LVE context. The VPS SSHes to Hostinger and holds the connection open, giving the plugin's engine an unlimited execution window.

Companion plugin PR: peptide-repo/prautoblogger#171 (v0.22.1 — adds --sync flag).

Alerting channel justification

System mail → cto@peptiderepo.com → Cloudflare Email Routing → peptiderepo@gmail.com. Chosen because: (1) zero external API tokens required on the VPS, (2) the Cloudflare routing is already live and tested, (3) simpler to operate than a Discord webhook (no token rotation), (4) delivery is reliable for low-frequency critical alerts (~1/day at most). Fallback: journal logs are always available via journalctl -u prab-generation -n 50.

Risk flags

DO NOT install or enable on the VPS this round. Code-only delivery; cutover is a gated, manual step documented in the runbook (see build report thread).
mail binary may not be installed on the VPS (apt-get install mailutils). The script degrades gracefully (|| log ...) — a missing mail does not abort the run.
SSH key at /root/.ssh/hostinger_prod_ed25519 must exist before production use. install-orchestrator.sh validates its presence.

Test plan

shellcheck infra/vps/orchestrate-generation.sh infra/vps/install-orchestrator.sh — expected: clean
DRY_RUN=1 infra/vps/orchestrate-generation.sh on the VPS — expected: echoes SSH commands, no generation
Post-v0.22.1-deploy: one real manual invocation from VPS (the 7-day catch-up run + self-test; see runbook)
Verify journal output: journalctl -u prab-generation -n 20
After successful manual run: systemctl enable --now prab-generation.timer, verify systemctl list-timers

HOLD FOR QA — do not merge.
Requires companion PR: peptide-repo/prautoblogger#171.

Agent-Session: prab-reliability-20260622

## What changed Adds `infra/vps/` — the VPS-side orchestrator for Option A (ADR 2026-06-22): the Coolify KVM8 VPS (76.13.220.15) drives PRAutoBlogger article generation on prod synchronously over a held-open SSH session, escaping Hostinger's ~10-min background-PHP-process kill. ### Files added **`infra/vps/orchestrate-generation.sh`** - Invokes `PRAutoBlogger_Run_Reaper::reap()` via `wp eval` at run start (belt-and-suspenders reaper sweep). - SSH to Hostinger and runs `wp prautoblogger generate --sync --path=...` (requires companion plugin PR v0.22.1). - Lock-aware: plugin's `Generation_Lock` DB mutex prevents double-drive; `--sync` exits 0 with `skipped` JSON if a live run is in progress. - Structured log lines to stdout (captured by `StandardOutput=journal` → `journalctl -u prab-generation`). - Failure alert: sends email to `cto@peptiderepo.com` via `mail` (forwards to peptiderepo@gmail.com via Cloudflare Email Routing). No external API token needed; system `mail` is the alert channel. - `DRY_RUN=1` mode: echoes SSH/WP-CLI commands without executing generation. - All config via env vars (`SSH_KEY`, `SSH_HOST`, `SSH_PORT`, `SSH_USER`, `WP_PATH`); no hardcoded secrets. - SSH key referenced by PATH (`/root/.ssh/hostinger_prod_ed25519`) — never embedded. - `shellcheck`-clean (`SC2029` disable on the SSH command with remote variable expansion). **`infra/vps/install-orchestrator.sh`** - One-time setup: copies orchestrator to `/opt/prab-orchestrator/`, installs `.service` + `.timer` to `/etc/systemd/system/`, runs `daemon-reload`. - **Does NOT enable or start the timer** — explicit post-QA cutover step. - Prints the exact cutover sequence for the operator. **`infra/vps/prab-generation.service`** - `Type=oneshot`, `User=root`, `TimeoutStartSec=7200` (covers 32-min LLM calls observed in prod). - `Restart=no` — timer retries next day on failure. - `SyslogIdentifier=prab-generation` for clean journal queries. **`infra/vps/prab-generation.timer`** - `OnCalendar=*-*-* 06:00:00 UTC` (fires before Hostinger daily wp-cron window). - `Persistent=true` — if VPS was down at 06:00 UTC, fires immediately on next boot. - `RandomizedDelaySec=300` — jitter for future multi-VPS scenarios. --- ## Why Hostinger's shared-hosting plan kills detached background PHP processes at ~10 min (CloudLinux LVE). A single LLM draft call observed 1925s (~32 min). SSH-attached processes survive because they run under a different LVE context. The VPS SSHes to Hostinger and holds the connection open, giving the plugin's engine an unlimited execution window. Companion plugin PR: peptide-repo/prautoblogger#171 (v0.22.1 — adds `--sync` flag). --- ## Alerting channel justification System `mail` → `cto@peptiderepo.com` → Cloudflare Email Routing → `peptiderepo@gmail.com`. Chosen because: (1) zero external API tokens required on the VPS, (2) the Cloudflare routing is already live and tested, (3) simpler to operate than a Discord webhook (no token rotation), (4) delivery is reliable for low-frequency critical alerts (~1/day at most). Fallback: journal logs are always available via `journalctl -u prab-generation -n 50`. --- ## Risk flags - **DO NOT install or enable on the VPS this round.** Code-only delivery; cutover is a gated, manual step documented in the runbook (see build report thread). - `mail` binary may not be installed on the VPS (`apt-get install mailutils`). The script degrades gracefully (`|| log ...`) — a missing `mail` does not abort the run. - SSH key at `/root/.ssh/hostinger_prod_ed25519` must exist before production use. `install-orchestrator.sh` validates its presence. --- ## Test plan 1. `shellcheck infra/vps/orchestrate-generation.sh infra/vps/install-orchestrator.sh` — expected: clean 2. `DRY_RUN=1 infra/vps/orchestrate-generation.sh` on the VPS — expected: echoes SSH commands, no generation 3. Post-v0.22.1-deploy: one real manual invocation from VPS (the 7-day catch-up run + self-test; see runbook) 4. Verify journal output: `journalctl -u prab-generation -n 20` 5. After successful manual run: `systemctl enable --now prab-generation.timer`, verify `systemctl list-timers` --- > **HOLD FOR QA — do not merge.** > Requires companion PR: peptide-repo/prautoblogger#171. Agent-Session: prab-reliability-20260622

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: peptide-repo/peptide-e2e#27