rdmsm4x crash recovery and session resumption

Recovered rdmsm4x after its 2026-08-30 watchdog panic, resumed only the provably interrupted work under preserved ownership, protected all dirty/untracked trees, restored StormGuard on all six fleet hosts, and produced a durable recovery report.

Scope

• Acting host: rdmsm4x.
• Fleet hosts verified: rdmsm4x, rdmbair13m5, rdmbair15m5, rdmpw3265m, rdmpw3275m, jdmbair13m5.
• Project lanes inspected: Tyrell, replicantDB, LogTTY, RDReceipt, ECS0/portfolio wiki, and fleet maintenance.
• Recovery task: Codex task 01a05374-5964-7343-96bd-832a026a5a00.

Changes made by this recovery task

1. Published /Users/richh/.agent-coordination/checkins/codex-rdmsm4x-crash-recovery-20260830.json to establish the recovery owner, target paths, resumed session identities, and preservation boundaries.
2. Sent immutable fleet-bus coordination messages, including:
   • 20260830-122041-F4037DB7: staging/worktree-protection acknowledgement.
   • 20260830-122957-D3140474: portfolio/ECS0 owner routing.
   • 20260830-123908-D6EA9C57: replicantDB double-writer warning.
   • 20260830-124353-21EA4B55: owner reconciliation for replicantDB, Tyrell, and StormGuard.
   • 20260830-124601-BA0C48CE: rdmpw3275m fleet-config-sync and TyrellBar discrepancies.
   • 20260830-125122-6443C115: urgent rdmbair15m5 StormGuard gap and high-load evidence.
3. Resumed original work rather than creating replacement owners:
   • RDReceipt Claude session 8dd75674-5b46-4367-9e01-a2b6baa3a83f in its original detached worktree.
   • Existing pinned Tyrell Codex task 01a04277-a949-77f1-8f07-57ff6a8ff90f; it ended executor-blocked without edits.
4. After rdmbair15m5 returned at approximately 99% CPU busy without StormGuard, ran the existing owner-authored, idempotent /Users/richh/dev/fleet/maintenance/scripts/host_storm_guard.zsh --deploy. This refreshed:
   • ~/.agent-coordination/storm_guard.zsh on all six hosts.
   • ~/Library/LaunchAgents/com.eastcoastscience.stormguard.plist on all six hosts.
   • The associated GUI LaunchAgent registration on all six hosts.
5. Added local Git ref refs/heads/recovery/rdreceipt-expense-export-20260830 pointing to detached RDReceipt commit aa942a22057b7b1fde5e7c380feb461707b1784b. The worktree remains detached, and its pre-existing untracked AGENTS.md remains untracked.
6. Wrote the user-facing report /Users/richh/Documents/Codex/2026-08-30/resu/outputs/rdmsm4x-crash-recovery-20260830.md.
7. Filed the changelog to Apple Notes. The installed helper's v2 behavior created a shared llmlog copy; because the supplied current fleet rule still requires a per-host folder, an additional verified copy was created under Notes folder rdmsm4x as rdmsm4x-changelog-20260830-1258-crash-recovery-and-session-resume. Neither copy was deleted or silently substituted for the other.

Changes made by resumed project owners

• RDReceipt session committed aa942a2 (Add date-range CSV and XLSX expense export) with these 11 paths:
  • CLAUDE.md
  • ISSUES.md
  • Packages/rdRECEIPT/Sources/rdRECEIPTCLI/main.swift
  • Packages/rdRECEIPT/Sources/rdRECEIPTCore/Models/ExportDateRange.swift
  • Packages/rdRECEIPT/Sources/rdRECEIPTUI/Export/CSVExportGenerator.swift
  • Packages/rdRECEIPT/Sources/rdRECEIPTUI/Export/XLSXExportGenerator.swift
  • Packages/rdRECEIPT/Sources/rdRECEIPTUI/Export/ZipArchiveWriter.swift
  • Packages/rdRECEIPT/Tests/rdRECEIPTCoreTests/ExportDateRangeTests.swift
  • Packages/rdRECEIPT/Tests/rdRECEIPTUITests/XLSXExportGeneratorTests.swift
  • SESSION-STATE.md
  • STATUS.md
• LogTTY owner committed clean 23f9a899c7d9024908cde31ad86c5254a63bd401 to advance the build number to 21.
• replicantDB owners advanced canonical main through 1.11.1–1.11.3 to the final observed e22009fee39701bd8bea169770821ee9d0fafcbb; one session closed the local 1.11.1 half-deploy and another is running an overnight memory curve. Source 1.11.3 reports 424 passing tests, while the local runtime deliberately remains 1.11.1 to preserve the measurement. No fleet fan-out was performed.
• ECS0/portfolio work advanced to 4c8ac836d3a5786040d5c453ef5f53c32ffcb078; its final observed 124-path dirty/untracked tree remains preserved for its project owner.

Commands and operations

• Read-only policy/state inspection: scutil, uptime, sysctl, ps, lsof, launchctl print, git status/log/show/worktree, tmutil status, vm_stat, top, panic-report inspection, task history inspection, and fleet-bus reads/syncs.
• Fleet identity and artifact checks used ssh -o BatchMode=yes to richh@<host>.ts.dataroo.net with bounded connect timeouts.
• A single Wake-on-LAN packet was sent through an online same-LAN peer. The target hardware address was streamed through encrypted SSH input and was neither printed nor persisted.
• The preserved topology-audit script was run read-only from a recovery copy because the canonical policy path is absent. Its stale account/project expectations were recorded rather than forced to pass.
• StormGuard deployment used its existing --deploy mode; no new remediation logic was written by this task.

Verification evidence

• Crash: /Library/Logs/DiagnosticReports/.contents.panic contains watchdog timeout: no checkins from watchdogd in 93 seconds; the record names pid 419 bluetoothd as the panicked task without proving root cause.
• StormGuard: SHA-256 a28207a40535b90f926e024febee9fb0e0a8824a20a651bf4e648c1c4ffa0b07 on all six hosts; each host returned its own ComputerName and a loaded com.eastcoastscience.stormguard LaunchAgent with last exit code 0.
• rdmbair15m5: pre-deploy top showed 99.2% CPU busy; first StormGuard pass showed 68% busy across 10 cores, 64% memory free, no swap, no kills, and no sweeps. Its second scheduled pass showed 51% busy, 60% memory free, no swap, and again no kills/sweeps; one-minute load fell to 9.45 by 12:59 EDT. Tyrell app/daemon PIDs were present and the app was signed universal arm64 + x86_64, Team ZU2882L4HT.
• RDReceipt: 534 tests / 89 suites / exit 0; openpyxl loaded typed XLSX dates/numbers with warnings treated as errors; Quick Look rendered; unzip -t passed; live-vault CLI range tests passed; temporary receipt-derived exports were deleted.
• RDReceipt preservation: show-ref --verify refs/heads/recovery/rdreceipt-expense-export-20260830 resolves to aa942a2; worktree HEAD is unchanged and detached; git status --short shows only ?? AGENTS.md.
• Tyrell task: executor negotiation failed three times, device probes showed both test iPhones unavailable, and the task reported no edits/build/install/deploy.
• LogTTY: canonical Git status clean at 23f9a899.

Backups and preservation

• No repository was reset, cleaned, stashed, force-pushed, or overwritten.
• Existing dirty/untracked project state was retained in place.
• The missing canonical topology audit was not reconstructed; three identical preserved copies remain under /Users/richh/dev/_migration/fleet-ingest/ with the observed hash prefix 5d1441ad.
• RDReceipt commit aa942a2 is now protected by the local branch recovery/rdreceipt-expense-export-20260830 in addition to its reflog.
• The Wake-on-LAN packet made no persistent configuration change.

Undo

• Delete the local RDReceipt recovery branch only after the commit is integrated elsewhere: git -C /Users/richh/dev/apps/RDReceipt branch -d recovery/rdreceipt-expense-export-20260830 (do not run while it is the only durable ref).
• StormGuard rollback requires booting out com.eastcoastscience.stormguard from each host's GUI domain and restoring/removing only the deployed script and plist under the fleet-maintenance owner. Do not roll it back while a host is under load without first preserving the evidence and confirming the owner.
• Fleet-bus messages and the recovery check-in are additive/immutable records and are intentionally not deleted.
• Project commits belong to their project owners and must not be reverted from this recovery lane.

Outstanding owner actions

• replicantDB: keep the scope split explicit while two Claude processes retain the same cwd; review the overnight RSS curve before any five-host fan-out. Do not use bare git push while the legacy upstream mapping remains unresolved.
• Tyrell: retry isolated CloudKit/chat QA only after the executor and iPhone transport are available.
• LogTTY: distinguish built, installed, and fleet-deployed build 21 with hashes.
• ECS0: accept the 123-path dirty tree and distinguish source commits from live publication.
• rdmpw3275m: repair/verify the fleet-config .local route and TyrellBar exit-78 state.
• Time Machine/NAS: investigate the slow rdmsm4x backup without killing backupd or discarding an in-flight backup.