rdmsm4x — login resume consolidation: twelve LaunchAgents to one capped dispatcher

2026-09-01 18:13 → 18:30 EDT · rdmsm4x · claude@rdmsm4x/1dc2a549-bfd5-4c40-b573-529f65e2d266

One line: a single login on 2026-08-31 opened 12 Terminal windows and 11 concurrent agent sessions and drove rdmsm4x — the fleet's ticket authority — to load 112 and ssh timeouts, taking the fleet's claim ledger with it; that is now one guarded dispatcher with a cap of 3 and a digest naming everything it deferred. ISSUE-20260831-21 and TASK-20260831-15 resolved.

Scope

rdmsm4x only. No other host was touched. The storm was rdmsm4x-only — claude@rdmpw3265m measured zero resume agents on its own host.

Why

Measured 2026-08-31 21:36–21:47 by claude@rdmpw3265m from outside the box: rdmsm4x rebooted at 21:36, twelve RunAtLoad LaunchAgents fired at login (nine Claude, one agy, two Codex), and 12 Terminal windows and 11 concurrent Claude sessions came up with nobody having typed anything. Load average 112. ssh into rdmsm4x hit a 180s timeout, so claude@rdmpw3275m could not file its work. Because rdmsm4x is the ticket authority, every other agent on every other host lost the ability to claim work for the duration — a coordination outage.

Nine of the twelve had no liveness guard, so a flapping login could stack duplicate windows onto live conversations. Nobody chose twelve; each session reasonably added one.

Rich chose option C then A — cap it, then one dispatcher.

What changed

Added

Moved, not deleted — 12 plists to ~/dev/fleet/ops/resume/archive/launchagents-retired-20260901/ with a restore README. launchctl bootout gui/501/<label> returned 0 for all twelve.

Deliberately untouched: every PINNED-SESSION* file, every target script (resume_pinned_on_login.zsh, resume_claude_dev73.zsh, resume_rdreceipt_on_login.zsh, resume_agy_on_login.zsh, both Codex launchers), and codex@rdmsm4x's pending marker under ~/Documents/Codex/2026-08-29/hel/. Pins point at owners' pin files rather than copying session ids, so re-pinning works exactly as each file's own header documents and no owner has to do anything.

How the dispatcher behaves

Per boot, in priority order, per pin: once-per-boot receipt → process match on the session id → transcript must exist → transcript mtime ≥ 4h → cap of 3, staggered 20s → dispatch → verify the process actually started (osascript rc=0 means Terminal accepted the command, not that anything ran).

The cap is not a silent loss. LAST-BOOT-DIGEST.md lists every deferred, live and stale pin with the exact command to resume it by hand.

Commands run

launchctl bootout gui/501/<label>          # x12, rc=0 each
mv <plist> ~/dev/fleet/ops/resume/archive/launchagents-retired-20260901/
launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.eastcoastscience.resume-dispatcher.plist
zsh ~/dev/fleet/ops/resume/resume_dispatcher.zsh --status

Verification

Counts, not "green" — a deletion is silent, a number is not.

• Login agents that resume a session or open Terminal: 12 → 1. Measured by reading ProgramArguments of every ~/Library/LaunchAgents/*.plist, never a filename glob — a claude-resume-* glob had missed four such agents under three naming conventions and produced the original count of 8. The only other Terminal-touching RunAtLoad agent is com.fleet.terminal-appearance, which sets appearance and starts no session.
• launchctl list | grep -i resume → only com.eastcoastscience.resume-dispatcher.
• Pins registered: 12. Archived plists: 12.
• Guard proven in both directions. Live sessions refused: 11 of 12. A genuinely dead session (temp pin at an 8-day-cold transcript, no process) still dispatched — no overshoot. A fabricated session id refused loudly as STALE.
• Cap holds: 5 eligible temp pins → 3 dispatched, 2 deferred with the manual command each.
• End-to-end dispatch proven on a throwaway pin set in scratch: Terminal window opened, log recorded VERIFIED: a process matching cc8f642b… is running (osascript rc=0).
• Per-boot receipt one-shot: second run same boot logged already dispatched for boot 1788226592; a foreign token released the gate.
• Loading the new agent fired it and it correctly no-opped on a pre-seeded receipt — no window opened on a box already holding 11 live sessions.

Not verified: an actual login dispatch. It cannot be observed before the next reboot. Read ~/dev/fleet/ops/resume/LAST-BOOT-DIGEST.md then.

Traps found (now in the code and the README)

1. A liveness guard keyed to a flag spelling missed a live session. --conversation <sid> found nothing because agy runs agy --conversation=<sid> with an equals sign. The dispatcher was about to open a second window onto a live conversation. Match the session id alone: over-matching declines to resume (safe), under-matching duplicates onto live work.
2. Transcript mtime is not a liveness signal for every agent. A live agy session had a transcript 20.5h cold. Both guards run because neither is load-bearing alone.
3. sysctl -n kern.boottime prints { sec = …, usec = … } — an unanchored .*sec = captures usec. It still changed every boot, so the receipt still worked and the bug would have sat there looking correct.
4. A 5-minute liveness threshold is wrong and fired for real on 2026-08-31, resuming a session that was alive and merely waiting for Rich. The threshold must clear a plausible think time.

Backup / undo

Nothing was deleted. To restore any retired agent:

mv ~/dev/fleet/ops/resume/archive/launchagents-retired-20260901/<label>.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$UID ~/Library/LaunchAgents/<label>.plist

Restoring an agent recreates the storm. To bring a session back, add or re-enable a pin.

To stop the dispatcher entirely: launchctl bootout gui/$UID/com.eastcoastscience.resume-dispatcher.

Outstanding, and explicitly not mine

• com.eastcoastscience.codex-resume-hel was already failing — launchctl list reported last exit 127 on 2026-08-31, before any of this. Cause unexplained; not fixed here. Its pin uses a cmd: override to run codex@rdmsm4x's own .command launcher verbatim rather than reimplementing its one-shot marker and seed prompt in a second place.
• Priorities in pins.d (dev, tyrell, fleetcc first) are a judgement, not a measurement. One edit to reorder; one edit to max_resume to change the cap.

Records

Commit 802dc36 in ~/dev/fleet. Tickets ISSUE-20260831-21 and TASK-20260831-15 resolved with evidence. No secrets in this record.