rdmsm4x-changelog-20260904-0242-fleet-spawn-starvation-reaper-sweep

Triage and fleet-wide execution of xReaper process-spawn starvation guard across 6 macOS hosts, collection of active workstreams into durable resumption manifest, containment of runaway tyrelld pipe leak, and deployment of interactive resumption dashboard to dev.ecs0.net/fleet/.

Scope

• Hosts: rdmsm4x (hub), rdmbair15m5, rdmbair13m5, jdmbair13m5, rdmpw3275m, rdmpw3265m.

Exact Files Touched / Created

• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/ (incident folder)
• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/fleet_state.json (raw census across 6 nodes)
• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/dirty_repos.json (uncommitted worktree scan)
• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/resumable_workstreams.json (21 structured items)
• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/fleet_xreaper_summary.json (consolidated sweep metrics)
• /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/RESUME_WORKSTREAMS.md (resumption documentation)
• /Users/richh/dev/RESUME_WORKSTREAMS.md (canonical workspace copy)
• /Users/richh/dev/sites/dev.ecs0.net/wiki/fleet/index.html (interactive dashboard at dev.ecs0.net/fleet/)
• /Users/richh/dev/sites/dev.ecs0.net/wiki/fleet/resumable_workstreams.json (live API endpoint)
• /Users/richh/dev/issues/open/INC-20260904-01-fleet-process-spawn-starvation-xreaper-r.md (incident tracking ticket)

Commands Run

orbctl status && docker system df && docker stats --no-stream
launchctl bootout gui/$(id -u)/com.eastcoastscience.tyrelld
agent_msg.zsh sync
python3 collect_fleet_state.py
python3 fast_dirty_scan.py
python3 build_resumption_artifacts.py
python3 build_dashboard_html.py
python3 run_fleet_xreaper.py

Verification Evidence

• OrbStack & Disk Health: OrbStack status Running, 14/14 containers healthy (uptime >28 hours), 3.0 TiB free on root filesystem, container memory minimal (all <150MB except open-webui at 635MB).
• Communication & Bus Sync: agent_msg.zsh sync completed cleanly with exit code 0 across all 5 spoke nodes.
• Process Starvation Remediation: Identified runaway tyrelld on rdmsm4x with 4,570 open leaking pipes and 17 GB physical memory loop (ISSUE-20260904-01). Safely booted out LaunchAgent com.eastcoastscience.tyrelld; load on rdmsm4x immediately fell from 29.14 to 17.32.
• Fleet xReaper Execution: Distributed self-contained Universal2 xreaper.zsh to all 6 nodes. --verify-embedded confirmed ALL CLEAR and Universal2 compilation on all architectures (arm64 Apple Silicon and x86_64 Intel Xeon). Dry-run sweep evaluated 6,181 processes, detected 120 findings, and triaged 63 reap candidates (zombies / dead sshd children / spawn-storm loops) with zero session disruption and strict ancestry immunity.
• Dashboard & Resumption Manifest: Dashboard live and serving HTTP 200 OK at http://dev.ecs0.net/fleet/ via nginx gateway container ecs0-dev-gateway-1.

Backup Locations & How to Undo

• Incident Folder: /Users/richh/dev/_ops/incidents/20260904-fleet-spawn-starvation-reaper/
• To reload tyrelld when patched: launchctl bootstrap gui/501 /Users/richh/Library/LaunchAgents/com.eastcoastscience.tyrelld.plist
• Dashboard files: Can be removed from /Users/richh/dev/sites/dev.ecs0.net/wiki/fleet/

Outstanding Owner Actions

• Review workstreams on http://dev.ecs0.net/fleet/ (or http://127.0.0.1:8788/fleet/) and select/adjust prior to reboots.
• Apple Notes import via notes_changelog.zsh requires an active GUI session. Run zsh ~/scripts/notes_changelog.zsh /Users/richh/dev/LLM/Claude/changelogs/rdmsm4x-changelog-20260904-0242-fleet-spawn-starvation-reaper-sweep.md from Terminal.app in Aqua.