Move private monitoring behind watchdog extension

This commit is contained in:
Meta Vens 2026-04-29 18:50:19 +01:00
commit f17c46a750
7 changed files with 48 additions and 128 deletions

View file

@ -70,13 +70,3 @@ VOIDBOT_OFFSITE_BACKUP_MAX_AGE_HOURS=36
VOIDBOT_HEALTHCHECK_NOTIFY_REPEAT_HOURS=12
VOIDBOT_HEALTHCHECK_TASK_NAME=VoidBot Operations Watchdog
VOIDBOT_HEALTHCHECK_INTERVAL_MINUTES=60
REPIXELIZER_MONITOR_ENABLED=true
REPIXELIZER_MONITOR_PUBLIC_BASE_URL=https://repixelizer.gamecult.org
REPIXELIZER_MONITOR_SSH_TARGET=
REPIXELIZER_MONITOR_SSH_IDENTITY_FILE=
REPIXELIZER_MONITOR_RECENT_MINUTES=15
REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT=1
REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT=1
REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT=3
REPIXELIZER_MONITOR_SLOW_JOB_SECONDS=45
REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT=3

View file

@ -26,7 +26,7 @@ Git history and smoke artifacts carry routine proof. `state/evidence.jsonl` shou
- Interaction-memory logic is no longer one thousand-line slab; event/tone analysis, shared constants, and profile synthesis are split across `packages/core/src/interaction-memory-*.ts`.
- State storage is no longer one persistence omnibus; `packages/core/src/state-storage.ts` is now a thin factory over domain-specific Postgres/file store modules plus bootstrap/migration helpers.
- Ops health, backup verification, offsite sync, and dashboard surfaces already exist.
- The operations watchdog now also carries Repixelizer fire signals: public `/api/health` checks plus recent `repixelizer-gui` journald pressure over SSH when configured, routed through the same owner DM alert path.
- The operations watchdog now supports ignored local extension checks, so adjacent-service fire signals can ride the same owner DM alert path without hardcoding private project gossip into the public repo.
- The new state/notes surfaces are now the continuity spine for future nontrivial work.
## Likely Next Bounded Move

View file

@ -86,7 +86,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh
- `scripts/start-voidbot-stack.ps1`
- stack bootstrap, health checks, fresh build, stale-process cleanup, bot/worker restart, runtime status emission.
- `scripts/check-voidbot-operations.ps1`
- watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and configured Repixelizer fire signals.
- watchdog for process liveness, Qdrant, Postgres, Ollama, Discord auth, backup freshness, offsite sync freshness, and optional ignored local extension checks.
- `scripts/backup-voidbot-state.ps1`
- local backup of Postgres, Qdrant snapshots, and RAG archives.
- `scripts/restore-voidbot-state.ps1`
@ -96,10 +96,7 @@ This note is the source-grounded description of how the live VoidBot stack is sh
- `scripts/voidbot-operations-dashboard-lib.ps1`
- dashboard rendering for the local ops surface.
When configured, the watchdog now also federates a small amount of adjacent-service pain into the same ops lane:
- public `GET /api/health` against Repixelizer
- recent `repixelizer-gui` journald pressure on Yggdrasil over SSH
- owner DM escalation for queue saturation, failed jobs, stale cancellations, and slow-job clusters
When present, an ignored local script at `.voidbot/private/check-voidbot-operations.local.ps1` can inject adjacent-service checks into the same ops lane. That keeps private infrastructure gossip out of tracked code while preserving one boring place for smoke alarms to land.
## Storage Boundaries

View file

@ -19,7 +19,7 @@ Use the persistent project-state surfaces to keep future work source-grounded an
- add a real restore-drill cadence instead of stopping at backup verification
- reduce dependence on interactive workstation logon where practical
- keep watchdog signals and dashboard diagnostics honest when dependencies are degraded
- keep adjacent-service checks deliberate so VoidBot does not quietly mutate into a generic infra panic funnel
- keep adjacent-service checks behind private local extensions so VoidBot does not quietly mutate into a generic infra panic funnel
### 3. Admin And Forensics Surfaces

View file

@ -405,6 +405,25 @@ function Invoke-RemoteLinuxCommand {
return [string]($response -join "`n")
}
function Invoke-WatchdogExtension {
param(
[Parameter(Mandatory = $true)]
[string] $Path,
[Parameter(Mandatory = $true)]
[hashtable] $Context
)
. $Path
$extensionCommand = Get-Command -Name "Invoke-VoidBotWatchdogExtension" -ErrorAction SilentlyContinue
if ($null -eq $extensionCommand) {
throw "Watchdog extension at $Path did not define Invoke-VoidBotWatchdogExtension."
}
& $extensionCommand -Context $Context
}
function Open-OwnerDmChannel {
param(
[Parameter(Mandatory = $true)]
@ -515,16 +534,9 @@ $offsiteRemoteWindowsDir = if ($config.ContainsKey("OFFSITE_BACKUP_REMOTE_WINDOW
$offsiteTarget = if ($config.ContainsKey("OFFSITE_BACKUP_SSH_TARGET")) { $config["OFFSITE_BACKUP_SSH_TARGET"] } else { "" }
$offsiteKeepLatest = if ($config.ContainsKey("OFFSITE_BACKUP_REMOTE_KEEP_LATEST")) { [int]$config["OFFSITE_BACKUP_REMOTE_KEEP_LATEST"] } else { 14 }
$offsiteLabel = if ($config.ContainsKey("OFFSITE_BACKUP_LABEL") -and -not [string]::IsNullOrWhiteSpace($config["OFFSITE_BACKUP_LABEL"])) { $config["OFFSITE_BACKUP_LABEL"] } else { "offsite-auto" }
$repixelizerMonitorEnabled = if ($config.ContainsKey("REPIXELIZER_MONITOR_ENABLED") -and -not [string]::IsNullOrWhiteSpace($config["REPIXELIZER_MONITOR_ENABLED"])) { [System.Convert]::ToBoolean($config["REPIXELIZER_MONITOR_ENABLED"]) } else { $true }
$repixelizerPublicBaseUrl = if ($config.ContainsKey("REPIXELIZER_MONITOR_PUBLIC_BASE_URL") -and -not [string]::IsNullOrWhiteSpace($config["REPIXELIZER_MONITOR_PUBLIC_BASE_URL"])) { $config["REPIXELIZER_MONITOR_PUBLIC_BASE_URL"].TrimEnd("/") } else { "https://repixelizer.gamecult.org" }
$repixelizerSshTarget = if ($config.ContainsKey("REPIXELIZER_MONITOR_SSH_TARGET")) { $config["REPIXELIZER_MONITOR_SSH_TARGET"] } else { "" }
$repixelizerSshIdentityFile = if ($config.ContainsKey("REPIXELIZER_MONITOR_SSH_IDENTITY_FILE")) { $config["REPIXELIZER_MONITOR_SSH_IDENTITY_FILE"] } else { "" }
$repixelizerRecentMinutes = if ($config.ContainsKey("REPIXELIZER_MONITOR_RECENT_MINUTES")) { [int]$config["REPIXELIZER_MONITOR_RECENT_MINUTES"] } else { 15 }
$repixelizerQueueFullFailCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT")) { [int]$config["REPIXELIZER_MONITOR_QUEUE_FULL_FAIL_COUNT"] } else { 1 }
$repixelizerJobFailedFailCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT")) { [int]$config["REPIXELIZER_MONITOR_JOB_FAILED_FAIL_COUNT"] } else { 1 }
$repixelizerStaleCancelWarnCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT")) { [int]$config["REPIXELIZER_MONITOR_STALE_CANCEL_WARN_COUNT"] } else { 3 }
$repixelizerSlowJobSeconds = if ($config.ContainsKey("REPIXELIZER_MONITOR_SLOW_JOB_SECONDS")) { [double]$config["REPIXELIZER_MONITOR_SLOW_JOB_SECONDS"] } else { 45.0 }
$repixelizerSlowJobWarnCount = if ($config.ContainsKey("REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT")) { [int]$config["REPIXELIZER_MONITOR_SLOW_JOB_WARN_COUNT"] } else { 3 }
$watchdogExtensionScriptRaw = if ($config.ContainsKey("VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT") -and -not [string]::IsNullOrWhiteSpace($config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"])) { $config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"] } else { ".voidbot/private/check-voidbot-operations.local.ps1" }
$watchdogExtensionScript = Resolve-ConfigPath -RepoRoot $repoRoot -Value $watchdogExtensionScriptRaw -Fallback ".voidbot/private/check-voidbot-operations.local.ps1"
$watchdogExtensionExplicit = $config.ContainsKey("VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT") -and -not [string]::IsNullOrWhiteSpace($config["VOIDBOT_HEALTHCHECK_EXTENSION_SCRIPT"])
$logCaptureStarted = Start-LogCapture -LogPath $logPath
$previousWatchdogState = if (Test-Path -LiteralPath $watchdogStatePath) {
try {
@ -958,108 +970,28 @@ $files = Get-ChildItem -LiteralPath $dir -Filter $pattern -File | Sort-Object La
}
}
Set-WatchdogStep -CurrentStep "repixelizer_checks" -Detail "Checking Repixelizer public health and recent service pressure."
if (-not $repixelizerMonitorEnabled) {
Add-Check -Name "repixelizer.monitor" -Status "passed" -Detail "Repixelizer monitor is disabled by REPIXELIZER_MONITOR_ENABLED."
} else {
Set-WatchdogStep -CurrentStep "extension_checks" -Detail "Running local watchdog extension checks."
if (Test-Path -LiteralPath $watchdogExtensionScript) {
try {
$health = Get-Json -Url "$repixelizerPublicBaseUrl/api/health"
if ([string]$health.status -eq "ok") {
Add-Check -Name "repixelizer.health" -Status "passed" -Detail "Repixelizer public health endpoint is ok." -Data @{
url = "$repixelizerPublicBaseUrl/api/health"
}
} else {
Add-Check -Name "repixelizer.health" -Status "failed" -Detail "Repixelizer public health returned status '$([string]$health.status)'." -Data @{
url = "$repixelizerPublicBaseUrl/api/health"
}
Invoke-WatchdogExtension -Path $watchdogExtensionScript -Context @{
RepoRoot = $repoRoot
StorageRoot = $storageRoot
Config = $config
RuntimeStatusPath = $runtimeStatusPath
HealthStatusPath = $statusPath
OffsiteStatusPath = $offsiteStatusPath
}
Add-Check -Name "watchdog.extension" -Status "passed" -Detail "Loaded local watchdog extension checks." -Data @{
path = $watchdogExtensionScript
}
} catch {
Add-Check -Name "repixelizer.health" -Status "failed" -Detail "Repixelizer public health check failed: $($_.Exception.Message)" -Data @{
url = "$repixelizerPublicBaseUrl/api/health"
Add-Check -Name "watchdog.extension" -Status "failed" -Detail "Local watchdog extension failed: $($_.Exception.Message)" -Data @{
path = $watchdogExtensionScript
}
}
if ([string]::IsNullOrWhiteSpace($repixelizerSshTarget)) {
Add-Check -Name "repixelizer.journal" -Status "warning" -Detail "Repixelizer journal pressure check skipped because REPIXELIZER_MONITOR_SSH_TARGET is blank."
} else {
try {
$recentWindow = [Math]::Max(1, $repixelizerRecentMinutes)
$since = "$recentWindow minutes ago"
$escapedSince = $since.Replace("'", "'\''")
$journalCommand = "journalctl -u repixelizer-gui --since '$escapedSince' --no-pager -o cat"
$journalText = Invoke-RemoteLinuxCommand -Target $repixelizerSshTarget -IdentityFile $repixelizerSshIdentityFile -Command $journalCommand
$journalLines = @($journalText -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) })
$queueFullLines = @($journalLines | Where-Object { $_ -match "repixelizer_queue_full" })
$failedLines = @($journalLines | Where-Object { $_ -match "repixelizer_job_failed" })
$staleCancelLines = @($journalLines | Where-Object { $_ -match "repixelizer_job_canceled" -and $_ -match "stopped heartbeating" })
$slowRunSeconds = @(
$journalLines | ForEach-Object {
if ($_ -match "repixelizer_job_completed" -and $_ -match "run_seconds=([0-9]+(?:\.[0-9]+)?)") {
[double]$Matches[1]
}
} | Where-Object { $_ -ge $repixelizerSlowJobSeconds }
)
if ($queueFullLines.Count -ge $repixelizerQueueFullFailCount) {
Add-Check -Name "repixelizer.queue_full" -Status "failed" -Detail "Repixelizer queue filled $($queueFullLines.Count) time(s) in the last $recentWindow minute(s)." -Data @{
threshold = $repixelizerQueueFullFailCount
recentMinutes = $recentWindow
}
} else {
Add-Check -Name "repixelizer.queue_full" -Status "passed" -Detail "No Repixelizer queue saturation above threshold in the last $recentWindow minute(s)." -Data @{
count = $queueFullLines.Count
threshold = $repixelizerQueueFullFailCount
recentMinutes = $recentWindow
}
}
if ($failedLines.Count -ge $repixelizerJobFailedFailCount) {
Add-Check -Name "repixelizer.job_failed" -Status "failed" -Detail "Repixelizer logged $($failedLines.Count) failed job(s) in the last $recentWindow minute(s)." -Data @{
threshold = $repixelizerJobFailedFailCount
recentMinutes = $recentWindow
}
} else {
Add-Check -Name "repixelizer.job_failed" -Status "passed" -Detail "No Repixelizer job failures above threshold in the last $recentWindow minute(s)." -Data @{
count = $failedLines.Count
threshold = $repixelizerJobFailedFailCount
recentMinutes = $recentWindow
}
}
if ($staleCancelLines.Count -ge $repixelizerStaleCancelWarnCount) {
Add-Check -Name "repixelizer.stale_cancellations" -Status "warning" -Detail "Repixelizer canceled $($staleCancelLines.Count) stale browser job(s) in the last $recentWindow minute(s)." -Data @{
threshold = $repixelizerStaleCancelWarnCount
recentMinutes = $recentWindow
}
} else {
Add-Check -Name "repixelizer.stale_cancellations" -Status "passed" -Detail "Repixelizer stale-browser cancellations are below threshold." -Data @{
count = $staleCancelLines.Count
threshold = $repixelizerStaleCancelWarnCount
recentMinutes = $recentWindow
}
}
if ($slowRunSeconds.Count -ge $repixelizerSlowJobWarnCount) {
Add-Check -Name "repixelizer.slow_jobs" -Status "warning" -Detail "Repixelizer had $($slowRunSeconds.Count) job(s) at or above $repixelizerSlowJobSeconds run seconds in the last $recentWindow minute(s)." -Data @{
thresholdCount = $repixelizerSlowJobWarnCount
thresholdSeconds = $repixelizerSlowJobSeconds
maxRunSeconds = [Math]::Round(($slowRunSeconds | Measure-Object -Maximum).Maximum, 3)
recentMinutes = $recentWindow
}
} else {
Add-Check -Name "repixelizer.slow_jobs" -Status "passed" -Detail "Repixelizer slow-job count is below threshold." -Data @{
count = $slowRunSeconds.Count
thresholdCount = $repixelizerSlowJobWarnCount
thresholdSeconds = $repixelizerSlowJobSeconds
recentMinutes = $recentWindow
}
}
} catch {
Add-Check -Name "repixelizer.journal" -Status "failed" -Detail "Repixelizer journal pressure check failed: $($_.Exception.Message)" -Data @{
target = $repixelizerSshTarget
recentMinutes = $repixelizerRecentMinutes
}
}
} elseif ($watchdogExtensionExplicit) {
Add-Check -Name "watchdog.extension" -Status "warning" -Detail "Configured watchdog extension script is missing." -Data @{
path = $watchdogExtensionScript
}
}

View file

@ -6,3 +6,4 @@
{"ts":"2026-04-26T15:55:00+01:00","type":"architecture","status":"ok","note":"Interaction-memory logic was split into event/tone analysis, shared constants, and profile-synthesis modules; behavior stayed stable, but one file trying to be both the ear and the therapist had become the likelier future failure surface."}
{"ts":"2026-04-26T16:25:00+01:00","type":"architecture","status":"ok","note":"State storage was split into a thin factory, bootstrap/migration helper, per-domain Postgres stores, and dedicated rate-limit stores; behavior stayed stable, but one persistence omnibus had become the likelier future failure surface."}
{"ts":"2026-04-29T17:36:43Z","type":"implementation","status":"ok","note":"Operations watchdog now checks Repixelizer public health plus recent Yggdrasil journald pressure signals and routes queue-full, job-failed, stale-cancel, and slow-job issues through the existing owner DM notification path."}
{"ts":"2026-04-29T18:20:00+01:00","type":"architecture","status":"ok","note":"Adjacent-service fire signals now load through an ignored local watchdog extension script instead of living in tracked VoidBot code, so the owner alert path stays useful without baking private cross-project monitoring into the public repo."}

View file

@ -21,7 +21,7 @@ constraints:
- Public traffic still rides one remote Qwen box plus local embedding and ops infrastructure.
- Owner flow and public flow share infrastructure but have different provider and safety boundaries.
- Postgres, Qdrant, Ollama, Docker, and Discord failures are operationally real; observability matters as much as features.
- The watchdog now also carries selected adjacent-service fire signals, so cross-project monitoring scope must stay intentional instead of turning into generic infra soup.
- Adjacent-service monitoring is allowed locally, but the public watchdog surface should stay generic and load private fire-signal logic through ignored extension hooks instead of hardcoded project gossip.
- Persistent project state must stay small enough to rehydrate from without becoming a second swamp.
invariants:
@ -54,7 +54,7 @@ data_flow:
active_subgoals:
- Make the new project-state surfaces the default continuity path for nontrivial VoidBot work.
- Keep public-lane limits, moderation, and provider behavior constrained enough that one 4060 Ti box does not get flayed for sport.
- Preserve boring observability around Postgres, Qdrant, Ollama, Docker, backups, watchdogs, and the newly federated Repixelizer fire signals.
- Preserve boring observability around Postgres, Qdrant, Ollama, Docker, backups, watchdogs, and any private local fire-signal extensions.
- Continue splitting oversized runtime files where concern boundaries are already obvious, starting with the remaining bot, worker, and memory-policy monarchies.
accepted_design:
@ -73,7 +73,7 @@ landed_baseline:
- Local and offsite backups, watchdogs, and the operations dashboard are live.
- Interaction memory and behavioral profiling are live.
- Source-grounding enforcement and tool-driven retrieval are live.
- The operations watchdog now monitors Repixelizer public health plus recent Yggdrasil journald pressure signals and routes serious queue or job failures through the existing owner DM notification path.
- The operations watchdog can now load ignored local extension checks, so adjacent-service fire signals can piggyback on the owner DM path without turning the public repo into cross-project gossip.
- The owner Codex lane is now split into orchestration, runtime/parsing, shared helpers, and rendering modules instead of one swollen provider file.
- The Discord bot lane is now split into a gateway shell plus dedicated command-handler and support helper modules instead of one swollen entrypoint file.
- Interaction-memory logic is now split into event/tone analysis and profile-synthesis modules instead of one slab trying to be both the ear and the therapist.
@ -88,5 +88,5 @@ rejected_paths:
reason: vibes produce plausible slop that sounds smart and lies with a straight face
current_status:
summary: VoidBot now has a real project-memory spine plus four legibility-hardening passes on major runtime organs. The owner Codex lane, Discord bot lane, interaction-memory logic, and state-storage persistence layer were all split into smaller seams, and the ops spine now also carries Repixelizer fire-signal monitoring through the existing watchdog and owner-notification path. The stack is still dense, but it no longer depends purely on transcript memory or one-file monarchies to stay coherent.
summary: VoidBot now has a real project-memory spine plus four legibility-hardening passes on major runtime organs. The owner Codex lane, Discord bot lane, interaction-memory logic, and state-storage persistence layer were all split into smaller seams, and the ops spine now supports ignored local watchdog extensions for adjacent fire signals without baking private infrastructure gossip into the public repo. The stack is still dense, but it no longer depends purely on transcript memory or one-file monarchies to stay coherent.
next_action: Keep using the canonical state files for re-entry, then continue decomposing the next oversized organ with the clearest concern boundaries, likely the MCP server or local LLM provider.