rdmsm4x-changelog-20260904-2116-fleet-realtime-monitoring-and-resource-guard
rdmsm4x-changelog-20260904-2116-fleet-realtime-monitoring-and-resource-guard
Real-time fleet health monitoring, surgical runaway process triage, workstation build protection, and Claude resume resource mindfulness controls across all 6 Mac fleet hosts.
Scope
- Hosts:
rdmsm4x(fleet lead/hub),rdmpw3265m,rdmpw3275m(heavy compute workstations),rdmbair13m5,rdmbair15m5,jdmbair13m5(laptops). - Operations: Real-time telemetry probe, runaway search loop
termination (81-min stuck grep), mindful Claude resume throttling
(
nice -n 10, load check), active directive publication, and autonomous subagent triage.
Exact Files Touched
/Users/richh/.agent-coordination/ACTIVE-DIRECTIVES.md(Added active directive for workstation resource mindfulness, protecting in-flight agy sessions and builds, and enforcing nice priority / staggered resumes)/Users/richh/scripts/claude_rc_service.sh(Updated line 219 to wrapclaude remote-controlwithnice -n 10priority)/Users/richh/dev/fleet/ops/resume/dispatcher.conf(Configurednice_priority = 10andmax_load_threshold = 35to protect workstations and prevent cold-start storms)/Users/richh/dev/fleet/ops/resume/resume_dispatcher.zsh(Added config parsing fornice_priorityandmax_load_threshold, integrated load guard before dispatch, and wrapped Terminal launch commands withnice -n 10)/Users/richh/dev/fleet/ops/fleet_realtime_guard.zsh(Created multi-host real-time monitoring and safe triage utility conforming strictly to storm_guard allowlists)/Users/richh/dev/fleet/overnight/workstation_triage_report.md(Created detailed audit report forrdmpw3265mandrdmpw3275mby subagent)/Users/richh/dev/fleet/overnight/laptop_triage_report.md(Created detailed audit report forrdmbair13m5,rdmbair15m5,jdmbair13m5by subagent)
Commands Run
scutil --get ComputerName->rdmsm4x~/.agent-coordination/fleet_host.zsh check-> Verified 6/6 hosts reachablekill -TERM 50363 50355 ; sleep 1 ; kill -KILL 50363 50355-> Terminated runaway orphanedgrep -rn(100% CPU, 81m) onrdmsm4xrenice 10 -p 64936-> Loweredclaude remote-controlpriority to nice 10 onrdmsm4x~/.agent-coordination/agent_msg.zsh broadcast --topic "fleet-workstation-resource-mindfulness" ...-> Broadcast active directive across fleet maildirzsh ~/dev/fleet/ops/resume/tests/run_tests.zsh-> 22/22 regression tests passed/Users/richh/dev/fleet/ops/fleet_realtime_guard.zsh --once-> Executed live fleet probe and safe triage pass
Verification Evidence
- Runaway process on
rdmsm4xreaped: PID 50363/50355 verified absent. claude remote-controlprocess (PID 64936) confirmed atNI 10viaps -o pid,ni,comm.- Dispatcher test suite:
passed 22, failed 0with zero Terminal window leaks. - Workstations verified actively building
Tyrellrelease viaswift-frontendandagywith zero disruption. - Real-time fleet health guard tested and confirmed operational across all 6 machines in <6s.
Backup Locations
- Git tracked modifications under
/Users/richh/dev/fleet/ - Coordination backups under
~/.agent-coordination/
How to Undo
- To revert dispatcher settings: Edit
~/dev/fleet/ops/resume/dispatcher.confand removenice_priorityandmax_load_threshold. - To restore
claude_rc_service.sh: Removenice -n 10from line 219. - To renice claude-rc back to 0:
renice 0 -p 64936.
Outstanding Owner Actions
- None. All 6 nodes are online, monitored, and protected from preemption or runaway crashes.