diff --git a/.env.example b/.env.example index e387a44..05908c2 100644 --- a/.env.example +++ b/.env.example @@ -70,13 +70,3 @@ VOIDBOT_OFFSITE_BACKUP_MAX_AGE_HOURS=36 VOIDBOT_HEALTHCHECK_NOTIFY_REPEAT_HOURS=12 VOIDBOT_HEALTHCHECK_TASK_NAME=VoidBot Operations Watchdog VOIDBOT_HEALTHCHECK_INTERVAL_MINUTES=60 -REPIXELIZER_MONITOR_ENABLED=true -REPIXELIZER_MONITOR_PUBLIC_BASE_URL=https://repixelizer.gamecult.org -REPIXELIZER_MONITOR_SSH_TARGET= -REPIXELIZER_MONITOR_SSH_IDENTITY_FILE= -REPIXELIZER_MONITOR_RECENT_MINUTES=15 -REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT=1 -REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT=1 -REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT=3 -REPIXELIZER_MONITOR_SLOW_JOB_SECONDS=45 -REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT=3 diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 0eae1b8..13bfa25 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -26,7 +26,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou - Interaction-memory logic is no longer one thousand-line slab; event/tone analysis, shared constants, and profile synthesis are split across `packages/core/src/interaction-memory-*.ts`. - State storage is no longer one persistence omnibus; `packages/core/src/state-storage.ts` is now a thin factory over domain-specific Postgres/file store modules plus bootstrap/migration helpers. - Ops health, backup verification, offsite sync, and dashboard surfaces already exist. -- The operations watchdog now also carries Repixelizer fire signals: public `/api/health` checks plus recent `repixelizer-gui` journald pressure over SSH when configured, routed through the same owner DM alert path. +- The operations watchdog now supports ignored local extension checks, so adjacent-service fire signals can ride the same owner DM alert path without hardcoding private project gossip into the public repo. - The new state/notes surfaces are now the continuity spine for future nontrivial work. ## Likely Next Bounded Move diff --git a/notes/voidbot-current-system-map.md b/notes/voidbot-current-system-map.md index 480f8de..da09a19 100644 --- a/notes/voidbot-current-system-map.md +++ b/notes/voidbot-current-system-map.md @@ -86,7 +86,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh - `scripts/start-voidbot-stack.ps1` - stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission. - `scripts/check-voidbot-operations.ps1` - - watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and configured Repixelizer fire signals. + - watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks. - `scripts/backup-voidbot-state.ps1` - local backup of Postgres, Qdrant snapshots, and RAG archives. - `scripts/restore-voidbot-state.ps1` @@ -96,10 +96,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh - `scripts/voidbot-operations-dashboard-lib.ps1` - dashboard rendering for the local ops surface. -When configured, the watchdog now also federates a small amount of adjacent-service pain into the same ops lane: -- public `GET /api/health` against Repixelizer -- recent `repixelizer-gui` journald pressure on Yggdrasil over SSH -- owner DM escalation for queue saturation, failed jobs, stale cancellations, and slow-job clusters +When present, an ignored local script at `.voidbot/private/check-voidbot-operations.local.ps1` can inject adjacent-service checks into the same ops lane. That keeps private infrastructure gossip out of tracked code while preserving one boring place for smoke alarms to land. ## Storage Boundaries diff --git a/notes/voidbot-implementation-plan.md b/notes/voidbot-implementation-plan.md index 7d890db..623df96 100644 --- a/notes/voidbot-implementation-plan.md +++ b/notes/voidbot-implementation-plan.md @@ -19,7 +19,7 @@ Use the persistent project-state surfaces to keep future work source-grounded an - add a real restore-drill cadence instead of stopping at backup verification - reduce dependence on interactive workstation logon where practical - keep watchdog signals and dashboard diagnostics honest when dependencies are degraded -- keep adjacent-service checks deliberate so VoidBot does not quietly mutate into a generic infra panic funnel +- keep adjacent-service checks behind private local extensions so VoidBot does not quietly mutate into a generic infra panic funnel ### 3. Admin And Forensics Surfaces diff --git a/scripts/check-voidbot-operations.ps1 b/scripts/check-voidbot-operations.ps1 index f461f17..746478a 100644 --- a/scripts/check-voidbot-operations.ps1 +++ b/scripts/check-voidbot-operations.ps1 @@ -405,6 +405,25 @@ function Invoke-RemoteLinuxCommand { return [string]($response -join "`n") } +function Invoke-WatchdogExtension { + param( + [Parameter(Mandatory = $true)] + [string] $Path, + [Parameter(Mandatory = $true)] + [hashtable] $Context + ) + + . $Path + + $extensionCommand = Get-Command -Name "Invoke-VoidBotWatchdogExtension" -ErrorAction SilentlyContinue + + if ($null -eq $extensionCommand) { + throw "Watchdog extension at $Path did not define Invoke-VoidBotWatchdogExtension." + } + + & $extensionCommand -Context $Context +} + function Open-OwnerDmChannel { param( [Parameter(Mandatory = $true)] @@ -515,16 +534,9 @@ $offsiteRemoteWindowsDir = if ($config.ContainsKey("OFFSITE_BACKUP_REMOTE_WINDOW $offsiteTarget = if ($config.ContainsKey("OFFSITE_BACKUP_SSH_TARGET")) { $config["OFFSITE_BACKUP_SSH_TARGET"] } else { "" } $offsiteKeepLatest = if ($config.ContainsKey("OFFSITE_BACKUP_REMOTE_KEEP_LATEST")) { [int]$config["OFFSITE_BACKUP_REMOTE_KEEP_LATEST"] } else { 14 } $offsiteLabel = if ($config.ContainsKey("OFFSITE_BACKUP_LABEL") -and -not [string]::IsNullOrWhiteSpace($config["OFFSITE_BACKUP_LABEL"])) { $config["OFFSITE_BACKUP_LABEL"] } else { "offsite-auto" } -$repixelizerMonitorEnabled = if ($config.ContainsKey("REPIXELIZER_MONITOR_ENABLED") -and -not [string]::IsNullOrWhiteSpace($config["REPIXELIZER_MONITOR_ENABLED"])) { [System.Convert]::ToBoolean($config["REPIXELIZER_MONITOR_ENABLED"]) } else { $true } -$repixelizerPublicBaseUrl = if ($config.ContainsKey("REPIXELIZER_MONITOR_PUBLIC_BASE_URL") -and -not [string]::IsNullOrWhiteSpace($config["REPIXELIZER_MONITOR_PUBLIC_BASE_URL"])) { $config["REPIXELIZER_MONITOR_PUBLIC_BASE_URL"].TrimEnd("/") } else { "https://repixelizer.gamecult.org" } -$repixelizerSshTarget = if ($config.ContainsKey("REPIXELIZER_MONITOR_SSH_TARGET")) { $config["REPIXELIZER_MONITOR_SSH_TARGET"] } else { "" } -$repixelizerSshIdentityFile = if ($config.ContainsKey("REPIXELIZER_MONITOR_SSH_IDENTITY_FILE")) { $config["REPIXELIZER_MONITOR_SSH_IDENTITY_FILE"] } else { "" } -$repixelizerRecentMinutes = if ($config.ContainsKey("REPIXELIZER_MONITOR_RECENT_MINUTES")) { [int]$config["REPIXELIZER_MONITOR_RECENT_MINUTES"] } else { 15 } -$repixelizerQueueFullFailCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT")) { [int]$config["REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT"] } else { 1 } -$repixelizerJobFailedFailCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT")) { [int]$config["REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT"] } else { 1 } -$repixelizerStaleCancelWarnCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT")) { [int]$config["REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT"] } else { 3 } -$repixelizerSlowJobSeconds = if ($config.ContainsKey("REPIXELIZER_MONITOR_SLOW_JOB_SECONDS")) { [double]$config["REPIXELIZER_MONITOR_SLOW_JOB_SECONDS"] } else { 45.0 } -$repixelizerSlowJobWarnCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT")) { [int]$config["REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT"] } else { 3 } +$watchdogExtensionScriptRaw = if ($config.ContainsKey("VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT") -and -not [string]::IsNullOrWhiteSpace($config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"])) { $config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"] } else { ".voidbot/private/check-voidbot-operations.local.ps1" } +$watchdogExtensionScript = Resolve-ConfigPath -RepoRoot $repoRoot -Value $watchdogExtensionScriptRaw -Fallback ".voidbot/private/check-voidbot-operations.local.ps1" +$watchdogExtensionExplicit = $config.ContainsKey("VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT") -and -not [string]::IsNullOrWhiteSpace($config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"]) $logCaptureStarted = Start-LogCapture -LogPath $logPath $previousWatchdogState = if (Test-Path -LiteralPath $watchdogStatePath) { try { @@ -958,108 +970,28 @@ $files = Get-ChildItem -LiteralPath $dir -Filter $pattern -File | Sort-Object La } } - Set-WatchdogStep -CurrentStep "repixelizer_checks" -Detail "Checking Repixelizer public health and recent service pressure." - if (-not $repixelizerMonitorEnabled) { - Add-Check -Name "repixelizer.monitor" -Status "passed" -Detail "Repixelizer monitor is disabled by REPIXELIZER_MONITOR_ENABLED." - } else { + Set-WatchdogStep -CurrentStep "extension_checks" -Detail "Running local watchdog extension checks." + if (Test-Path -LiteralPath $watchdogExtensionScript) { try { - $health = Get-Json -Url "$repixelizerPublicBaseUrl/api/health" - if ([string]$health.status -eq "ok") { - Add-Check -Name "repixelizer.health" -Status "passed" -Detail "Repixelizer public health endpoint is ok." -Data @{ - url = "$repixelizerPublicBaseUrl/api/health" - } - } else { - Add-Check -Name "repixelizer.health" -Status "failed" -Detail "Repixelizer public health returned status '$([string]$health.status)'." -Data @{ - url = "$repixelizerPublicBaseUrl/api/health" - } + Invoke-WatchdogExtension -Path $watchdogExtensionScript -Context @{ + RepoRoot = $repoRoot + StorageRoot = $storageRoot + Config = $config + RuntimeStatusPath = $runtimeStatusPath + HealthStatusPath = $statusPath + OffsiteStatusPath = $offsiteStatusPath + } + Add-Check -Name "watchdog.extension" -Status "passed" -Detail "Loaded local watchdog extension checks." -Data @{ + path = $watchdogExtensionScript } } catch { - Add-Check -Name "repixelizer.health" -Status "failed" -Detail "Repixelizer public health check failed: $($_.Exception.Message)" -Data @{ - url = "$repixelizerPublicBaseUrl/api/health" + Add-Check -Name "watchdog.extension" -Status "failed" -Detail "Local watchdog extension failed: $($_.Exception.Message)" -Data @{ + path = $watchdogExtensionScript } } - - if ([string]::IsNullOrWhiteSpace($repixelizerSshTarget)) { - Add-Check -Name "repixelizer.journal" -Status "warning" -Detail "Repixelizer journal pressure check skipped because REPIXELIZER_MONITOR_SSH_TARGET is blank." - } else { - try { - $recentWindow = [Math]::Max(1, $repixelizerRecentMinutes) - $since = "$recentWindow minutes ago" - $escapedSince = $since.Replace("'", "'\''") - $journalCommand = "journalctl -u repixelizer-gui --since '$escapedSince' --no-pager -o cat" - $journalText = Invoke-RemoteLinuxCommand -Target $repixelizerSshTarget -IdentityFile $repixelizerSshIdentityFile -Command $journalCommand - $journalLines = @($journalText -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) - $queueFullLines = @($journalLines | Where-Object { $_ -match "repixelizer_queue_full" }) - $failedLines = @($journalLines | Where-Object { $_ -match "repixelizer_job_failed" }) - $staleCancelLines = @($journalLines | Where-Object { $_ -match "repixelizer_job_canceled" -and $_ -match "stopped heartbeating" }) - $slowRunSeconds = @( - $journalLines | ForEach-Object { - if ($_ -match "repixelizer_job_completed" -and $_ -match "run_seconds=([0-9]+(?:\.[0-9]+)?)") { - [double]$Matches[1] - } - } | Where-Object { $_ -ge $repixelizerSlowJobSeconds } - ) - - if ($queueFullLines.Count -ge $repixelizerQueueFullFailCount) { - Add-Check -Name "repixelizer.queue_full" -Status "failed" -Detail "Repixelizer queue filled $($queueFullLines.Count) time(s) in the last $recentWindow minute(s)." -Data @{ - threshold = $repixelizerQueueFullFailCount - recentMinutes = $recentWindow - } - } else { - Add-Check -Name "repixelizer.queue_full" -Status "passed" -Detail "No Repixelizer queue saturation above threshold in the last $recentWindow minute(s)." -Data @{ - count = $queueFullLines.Count - threshold = $repixelizerQueueFullFailCount - recentMinutes = $recentWindow - } - } - - if ($failedLines.Count -ge $repixelizerJobFailedFailCount) { - Add-Check -Name "repixelizer.job_failed" -Status "failed" -Detail "Repixelizer logged $($failedLines.Count) failed job(s) in the last $recentWindow minute(s)." -Data @{ - threshold = $repixelizerJobFailedFailCount - recentMinutes = $recentWindow - } - } else { - Add-Check -Name "repixelizer.job_failed" -Status "passed" -Detail "No Repixelizer job failures above threshold in the last $recentWindow minute(s)." -Data @{ - count = $failedLines.Count - threshold = $repixelizerJobFailedFailCount - recentMinutes = $recentWindow - } - } - - if ($staleCancelLines.Count -ge $repixelizerStaleCancelWarnCount) { - Add-Check -Name "repixelizer.stale_cancellations" -Status "warning" -Detail "Repixelizer canceled $($staleCancelLines.Count) stale browser job(s) in the last $recentWindow minute(s)." -Data @{ - threshold = $repixelizerStaleCancelWarnCount - recentMinutes = $recentWindow - } - } else { - Add-Check -Name "repixelizer.stale_cancellations" -Status "passed" -Detail "Repixelizer stale-browser cancellations are below threshold." -Data @{ - count = $staleCancelLines.Count - threshold = $repixelizerStaleCancelWarnCount - recentMinutes = $recentWindow - } - } - - if ($slowRunSeconds.Count -ge $repixelizerSlowJobWarnCount) { - Add-Check -Name "repixelizer.slow_jobs" -Status "warning" -Detail "Repixelizer had $($slowRunSeconds.Count) job(s) at or above $repixelizerSlowJobSeconds run seconds in the last $recentWindow minute(s)." -Data @{ - thresholdCount = $repixelizerSlowJobWarnCount - thresholdSeconds = $repixelizerSlowJobSeconds - maxRunSeconds = [Math]::Round(($slowRunSeconds | Measure-Object -Maximum).Maximum, 3) - recentMinutes = $recentWindow - } - } else { - Add-Check -Name "repixelizer.slow_jobs" -Status "passed" -Detail "Repixelizer slow-job count is below threshold." -Data @{ - count = $slowRunSeconds.Count - thresholdCount = $repixelizerSlowJobWarnCount - thresholdSeconds = $repixelizerSlowJobSeconds - recentMinutes = $recentWindow - } - } - } catch { - Add-Check -Name "repixelizer.journal" -Status "failed" -Detail "Repixelizer journal pressure check failed: $($_.Exception.Message)" -Data @{ - target = $repixelizerSshTarget - recentMinutes = $repixelizerRecentMinutes - } - } + } elseif ($watchdogExtensionExplicit) { + Add-Check -Name "watchdog.extension" -Status "warning" -Detail "Configured watchdog extension script is missing." -Data @{ + path = $watchdogExtensionScript } } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 56261a6..121c220 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -6,3 +6,4 @@ {"ts":"2026-04-26T15:55:00+01:00","type":"architecture","status":"ok","note":"Interaction-memory logic was split into event/tone analysis, shared constants, and profile-synthesis modules; behavior stayed stable, but one file trying to be both the ear and the therapist had become the likelier future failure surface."} {"ts":"2026-04-26T16:25:00+01:00","type":"architecture","status":"ok","note":"State storage was split into a thin factory, bootstrap/migration helper, per-domain Postgres stores, and dedicated rate-limit stores; behavior stayed stable, but one persistence omnibus had become the likelier future failure surface."} {"ts":"2026-04-29T17:36:43Z","type":"implementation","status":"ok","note":"Operations watchdog now checks Repixelizer public health plus recent Yggdrasil journald pressure signals and routes queue-full, job-failed, stale-cancel, and slow-job issues through the existing owner DM notification path."} +{"ts":"2026-04-29T18:20:00+01:00","type":"architecture","status":"ok","note":"Adjacent-service fire signals now load through an ignored local watchdog extension script instead of living in tracked VoidBot code, so the owner alert path stays useful without baking private cross-project monitoring into the public repo."} diff --git a/state/map.yaml b/state/map.yaml index 12d5111..566ce7d 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -21,7 +21,7 @@ constraints: - Public traffic still rides one remote Qwen box plus local embedding and ops infrastructure. - Owner flow and public flow share infrastructure but have different provider and safety boundaries. - Postgres, Qdrant, Ollama, Docker, and Discord failures are operationally real; observability matters as much as features. - - The watchdog now also carries selected adjacent-service fire signals, so cross-project monitoring scope must stay intentional instead of turning into generic infra soup. + - Adjacent-service monitoring is allowed locally, but the public watchdog surface should stay generic and load private fire-signal logic through ignored extension hooks instead of hardcoded project gossip. - Persistent project state must stay small enough to rehydrate from without becoming a second swamp. invariants: @@ -54,7 +54,7 @@ data_flow: active_subgoals: - Make the new project-state surfaces the default continuity path for nontrivial VoidBot work. - Keep public-lane limits, moderation, and provider behavior constrained enough that one 4060 Ti box does not get flayed for sport. - - Preserve boring observability around Postgres, Qdrant, Ollama, Docker, backups, watchdogs, and the newly federated Repixelizer fire signals. + - Preserve boring observability around Postgres, Qdrant, Ollama, Docker, backups, watchdogs, and any private local fire-signal extensions. - Continue splitting oversized runtime files where concern boundaries are already obvious, starting with the remaining bot, worker, and memory-policy monarchies. accepted_design: @@ -73,7 +73,7 @@ landed_baseline: - Local and offsite backups, watchdogs, and the operations dashboard are live. - Interaction memory and behavioral profiling are live. - Source-grounding enforcement and tool-driven retrieval are live. - - The operations watchdog now monitors Repixelizer public health plus recent Yggdrasil journald pressure signals and routes serious queue or job failures through the existing owner DM notification path. + - The operations watchdog can now load ignored local extension checks, so adjacent-service fire signals can piggyback on the owner DM path without turning the public repo into cross-project gossip. - The owner Codex lane is now split into orchestration, runtime/parsing, shared helpers, and rendering modules instead of one swollen provider file. - The Discord bot lane is now split into a gateway shell plus dedicated command-handler and support helper modules instead of one swollen entrypoint file. - Interaction-memory logic is now split into event/tone analysis and profile-synthesis modules instead of one slab trying to be both the ear and the therapist. @@ -88,5 +88,5 @@ rejected_paths: reason: vibes produce plausible slop that sounds smart and lies with a straight face current_status: - summary: VoidBot now has a real project-memory spine plus four legibility-hardening passes on major runtime organs. The owner Codex lane, Discord bot lane, interaction-memory logic, and state-storage persistence layer were all split into smaller seams, and the ops spine now also carries Repixelizer fire-signal monitoring through the existing watchdog and owner-notification path. The stack is still dense, but it no longer depends purely on transcript memory or one-file monarchies to stay coherent. + summary: VoidBot now has a real project-memory spine plus four legibility-hardening passes on major runtime organs. The owner Codex lane, Discord bot lane, interaction-memory logic, and state-storage persistence layer were all split into smaller seams, and the ops spine now supports ignored local watchdog extensions for adjacent fire signals without baking private infrastructure gossip into the public repo. The stack is still dense, but it no longer depends purely on transcript memory or one-file monarchies to stay coherent. next_action: Keep using the canonical state files for re-entry, then continue decomposing the next oversized organ with the clearest concern boundaries, likely the MCP server or local LLM provider.