rdmsm4x-changelog-20260926-1938-halted-watcher

Built and installed a no-LLM halted-work detector; full canonical/gate delivery remains owner-gated.

Halted watcher result — TASK-20260926-69

Owner Operations Grok Bot, ops@rdmsm4x. Executor codex@rdmsm4x/halted0926. 2026-09-26. Production. Host verified rdmsm4x.

Result and delivery boundary

The no-LLM detector is built, tested, pushed to the fleet remote and installed as com.eastcoastscience.halted in gui/501. Its launchd run exited 0. It currently runs from the isolated worktree, NOT the canonical fleet checkout. Full canonical integration and live bot wake are NOT complete; see named gates below. No halted session was answered, resumed, or sent keys. No existing watcher was retired.

Reuse

Adapted pure pattern/redaction helpers from antistall-c1-c9/watcher/stall_scan.py. Avoided importing its module because import performs a provider transcript sweep.
Reused botwake_common frontmatter parser, event validation, Ops routing and the unchanged wake policy/caps. Added source halted and ops tag rule.
Read herdr-agent-state.sh v8: it reports session identity, not waiting reason. Existing Claude Stop hooks are project-state and antistall; Notification absent. Herdr runtime agent list/read provides four useful additional pane observations.
Reviewed directive-resolve-dont-hand-back, resume and watch. Policy propagation, login resumption and worktree preservation remain their own responsibilities.

Built

ops/halted/: matcher, watcher, hook collector, JSON/TOML hook installer and field-scoped rollback, uv-managed Python launcher, launchd installer/plist builder, one-step rollback, fixtures/tests and living docs. Exact changed paths are in CHANGED-FILES.txt. Source lives at ~/dev/_worktrees/halted0926/ops/halted.

Default tmux server panes: capture final ~40 lines and inspect foreground CLI basenames, never argv. Prompt/permission/menu/question patterns plus 20-minute unchanged input; supplement local herdr panes. Scan waiting ticket statuses/tags, including observed needs-decision. Hooks append sanitized JSONL, expire after two hours, and are superseded by real clear/pane evidence. No provider transcript crawl.

Atomic digest JSON/Markdown includes entity, host, agent, redacted question, first_seen and class guess. Persistent pane hashes and seen IDs suppress repeats. An unchanged/empty successful run exits 0 silently. A planted fake event and permission prompt must both detect on every run; broken control/source -> exit 2. Heartbeat records coverage and explicitly reports hooks_configured=false today.

New halt -> deterministic botwake.event/1 JSONL, source halted, severity high, target ops. Local inbox spools remain until durable held/simulated/sent receipt. Both gate orchestrators acknowledge safely; gate WatchPaths includes the inbox, covering 00:50 without waiting for the morning calendar. Producers never call webhooks or bypass caps. Adapter remains staged until owner integration.

Schedule: 16 calendar entries, hourly :50 for hours 09–23 and 00, ET. Nice 10, ProcessType Background, low-priority I/O. No cron/nohup/LLM in the runtime.

Commits and publication

39ec2d35c1c3f40246fe7a557b6681e556e92b35 — initial watcher/inbox implementation.
e0a0004bb46fc55134883fa32ba48ba345865006 — hook/rollback hardening, review fixes, gate inbox WatchPaths, final docs.
Remote: fleet (git.ecs0.net:git/root/fleet.git), branch codex/halted0926.
git ls-remote fleet refs/heads/codex/halted0926 returned the exact e0a0004 SHA.
Worktree status clean at final check. Canonical root dirty edits were preserved.

Tests and evidence

Re-run:

uv run --no-project --offline --python 3.14 -m unittest discover -s /Users/richh/dev/_worktrees/halted0926/ops/halted/tests -v
uv run --no-project --offline --python 3.14 -m unittest discover -s /Users/richh/dev/_worktrees/halted0926/ops/botwake/tests -v

39 watcher tests, 0 failures (tests.log).
34 botwake tests, 0 failures (botwake-tests.log).
Independent Luna reviewer reproduced three defects, verified fixes and freshly passed 38 watcher + 34 gate tests before the final WatchPaths regression was added. REVIEW.md records the exact evidence and limits.
Fixed: idle Notification clearing pending Stop; plain numbered menu omission; exact field rollback after partial installation failure.
Real captured working/completed negatives; explicitly planted captured positive. No naturally halted CLI pane was available during initial sampling. This does not prove recall for every possible CLI UI.
Plist lint and zsh syntax checks pass.
FIRST-DIGEST.md: 13 waiting tickets, 12/12 tmux panes scanned, four herdr panes, zero halted CLI panes on that pass. FIRST-HEARTBEAT.json records it.
Real 13-event replay into isolated dry-run wake layer -> one ops would-wake; gate-real-digest-proof.json. No live webhook in this proof.
Live unchanged repeat rc0, stdout 0 bytes, stderr 0 bytes (unchanged-run.*).
launchd-proof.txt: loaded job, nice=10, last exit code=0. The kickstart was executed through launchd; natural hourly/calendar rollover not yet observed.

Schedules and duplicates

SCHEDULE-INVENTORY.md/.json lists all 28 related installed plists; loaded-state capture in related-launchd-loaded.txt. User crontab empty; root has no crontab (root-crontab-inventory.txt). DEC-20260926-13 resolved RETAIN BOTH: grok-stall-watch has transcript coverage omitted here, so a full duplicate is not established. No schedule/job retired. Only this new halted job installed.

Rollback, actually tested

zsh /Users/richh/dev/_worktrees/halted0926/ops/halted/rollback.zsh

rollback-proof.log records rc0, verified absent from launchctl, plist archived to ~/archive/halted-20260926-192725-32911/. Reinstall succeeded (reinstall.log). The job is loaded again. After canonical promotion the same command is available at ~/dev/fleet/ops/halted/rollback.zsh. Rollback preserves digests, event evidence, and already queued gate events; it does not change botwake mode or other jobs. If hooks are later installed, the same rollback removes only their entries and restores the previous notify command, refusing an unrelated peer replacement.

Named gates and exact continuation

Canonical integration, owner lane: worker isolation prohibits overwriting the live dirty fleet checkout. Ops/Claude must integrate fleet/codex/halted0926, preserve dirty botwake rules/wake edits, and install from ~/dev/fleet/ops/halted/install.zsh. Reload the gate plist with its added inbox WatchPaths using existing botwake/install.zsh --apply. Requests delivered on bus 20260926-191554-D9AECB75, 192321-9F4576A0, 192917-18E60BDF; acceptance not yet received. Do not mistake branch publication for main integration.
DEC-20260926-12, Claude approval: global hooks are implemented and tested in sandbox homes but NOT installed. JSON/TOML preservation and effortLevel pins tested. Explicit Claude sign-off requested via bus (including addressed live Claude identities at 19:31). On approval, run uv-managed configure_hooks.py install --approved-dec DEC-20260926-12, then prove Notification/Stop and chained Codex notify locally. Never use plugin enable/disable.
DEC-20260926-15, ops credential: gate currently dry-run, LIVE-APPROVED exists, Keychain service botwake.webhook.ops absent, no matching global.env key name. No live wake can occur. Store the owner's webhook directly in the approved Keychain service/account per gate README, then enable approved live mode and prove a real ops wake. No secret value belongs in a ticket, argv or this handoff.
Apple Notes: this harness is Background, so notes_changelog refuses GUI AppleEvents. File copy retained; Notes publication pending, not claimed saved.

Existing topology preflight path was stale; discovered equivalent under fleet/dev-fleet-reconciliation/scripts and ran it. It checked six hosts and returned three existing topology findings (topology-audit.log), outside this local watcher scope; no topology/auth state was changed.

Budget: usage_status advice=proceed before one bounded Luna review. No paid service, new subscription, or runtime LLM. Reviewer spend in tokens/dollars not exposed.

Exact local state paths

ops/botwake/INTERFACE.md ops/botwake/bot_wake_gate.py ops/botwake/bot_wake_gate.zsh ops/botwake/botwake_common.py ops/botwake/botwake_inbox.py ops/botwake/botwake_watch.py ops/botwake/com.eastcoastscience.botwake.gate.plist ops/botwake/event.schema.json ops/botwake/install.zsh ops/botwake/rules.json ops/halted/.gitignore ops/halted/AGENTS.md ops/halted/DECISIONS.md ops/halted/INTERFACES.md ops/halted/ISSUES.md ops/halted/README.md ops/halted/SESSION-STATE.md ops/halted/STATUS.md ops/halted/configure_hooks.py ops/halted/hook.py ops/halted/install.zsh ops/halted/matcher.py ops/halted/plist.py ops/halted/rollback.zsh ops/halted/run.zsh ops/halted/tests/fixtures/herdr-completed.json ops/halted/tests/fixtures/herdr-verified-completed.json ops/halted/tests/fixtures/tmux-planted.json ops/halted/tests/fixtures/tmux-working.json ops/halted/tests/test_halted.py ops/halted/watcher.py

Runtime: ~/.agent-coordination/halted/{digest.json,digest.md,state.json,heartbeat.json,botwake-events.jsonl,run.lock,launchd.out.log,launchd.err.log}; ~/.agent-coordination/checks/halted.json; ~/.agent-coordination/botwake/inbox/*.jsonl; ~/Library/LaunchAgents/com.eastcoastscience.halted.plist; PROJECTS.md additive entry; ticket/checkin/bus records listed above. Global Claude/Codex config untouched.