rdmsm4x-changelog-20260904-2122-fleet-load-shedding-canary-guard-and-slowdown-enforced

Remediated high system load by stopping ReplicantDB canary daemons and runaway test runners across the fleet, broadcasting agent slowdown directives, and deploying an automated load-shedding LaunchAgent guard.

Scope

• Hosts: rdmsm4x (canonical coordinator), rdmbair15m5, rdmbair13m5, jdmbair13m5, rdmpw3265m, rdmpw3275m
• Subsystems: Process lifecycle management, launchd services, fleet coordination message bus, system load monitoring

Exact Files Touched

• ~/scripts/fleet_load_canary_guard.py: Automated multi-node load inspector, canary/test app terminator, and slowdown broadcast guard.
• ~/Library/LaunchAgents/com.eastcoastscience.fleet-load-guard.plist: LaunchAgent executing fleet_load_canary_guard.py every 90 seconds.
• ~/.agent-coordination/fleet-load-guard-status.json: Machine-readable fleet load and remediation status.
• ~/.agent-coordination/fleet-load-guard-state.json: Historical state and broadcast rate-limit tracking.
• ~/Library/Logs/fleet-load-guard.log: Guard activity execution log.
• ~/.agent-coordination/mail/20260904-211945-75E5A408__from-agent-rdmsm4x__to-all__fleet-load-shedding-slowdown-directive.md: Fleet broadcast directive instructing all agents to slow down.

Commands Run

• launchctl bootout gui/$(id -u)/com.eastcoastscience.replicantDB.daemon on rdmsm4x, rdmbair13m5, rdmpw3265m, rdmpw3275m.
• Terminated canary daemon PIDs: 7283 (rdmsm4x), 60854 (rdmbair13m5), 49428 (rdmpw3265m), 9386 (rdmpw3275m).
• Intercepted and stopped runaway test runners: PID 88197 and PID 90362 (swift-test on rdmsm4x).
• agent_msg.zsh send --to all --topic fleet-load-shedding-slowdown-directive ...
• Loaded LaunchAgent com.eastcoastscience.fleet-load-guard.

Verification Evidence

• Zero ReplicantDB canary daemons active across all 6 fleet machines (verified by per-node ps).
• Zero runaway test runner processes active across all 6 fleet machines.
• User GUI applications (Rich's ReplicantDB GUI PID 63422, Claude.app, Xcode, Ghostty) preserved intact with zero interruption.
• Immediate load recovery across spokes:
  • rdmbair13m5: dropped from 7.00 to 4.35 (10 cores).
  • rdmpw3265m: dropped from 32.61 to 22.99 (48 cores).
  • rdmbair15m5: 4.13 (10 cores).
  • jdmbair13m5: 1.90 (10 cores).
• LaunchAgent com.eastcoastscience.fleet-load-guard running actively (PID 90410), logging clean cycles to ~/Library/Logs/fleet-load-guard.log.

Backup Locations & Undo

• Re-enable ReplicantDB daemon when load stabilizes: launchctl load ~/Library/LaunchAgents/com.eastcoastscience.replicantDB.daemon.plist (on desired host).
• Disable automated load guard: launchctl unload ~/Library/LaunchAgents/com.eastcoastscience.fleet-load-guard.plist