Fleet changelogs · dev.ecs0.net
rdmbair15m5-changelog-20260823-0243-agy-quota-hold-automated-resume

agy quota hold — automated resume armed for a 3-hour window

Host: rdmbair15m5 · Session: Claude Code richh-69 · Time: 2026-08-23 02:37 → 02:43 EDT

What happened

Rich flagged agy ttys001 hitting Individual quota reached … Resets in 3h3m28s. Surveying every agy tab showed two sessions blocked, not one:

tty state banner was doing
ttys000 blocked Resets in 3h1m10s 22 subagents in flight (teamwork_preview auditor/challenger running swift tests)
ttys001 blocked Resets in 3h3m28s fleet architecture diagram + fleet master LLM blueprint changelogs; grepping for "oneroute"
ttys003 / ttys004 / ttys007 / ttys008 no quota banner — unaffected

ttys001 shows a single banner, not a retry storm — it stopped cleanly after its last successful command at 02:36:03.

Deadlines, derived rather than guessed

The banner is a countdown, not a timestamp, so the wall-clock reset had to be computed from when it was observed. Parsed each tab's own countdown at read time:

ttys000  retry after 05:42:42 EDT
ttys001  retry after 05:45:00 EDT

Automated resume — survives this chat session and machine sleep

Three hours is far too long to hold in a conversation, so the resume is a LaunchAgent, not a loop in this session.

~/.agent-coordination/agy_quota_watch.zsh + net.dataroo.agyquotawatch (StartInterval 300, RunAtLoad):

  1. Enumerates Terminal tabs, keeps only agy ones, and skips any tab that is busy — it will never interrupt working work.
  2. On a fresh quota hit (a new Error ID), parses that tab's own countdown and stores a deadline.
  3. From 60s before the deadline, sends a resume prompt, then a bare Return — long pastes arrive in agy's input box unsubmitted, which cost three resumes earlier tonight.
  4. Confirms success by the Error ID technique: every quota failure prints a unique Error ID: <uuid>-<n>. If a new id appears after the send it is still blocked and it retries; if the id is unchanged the turn was accepted. Far more reliable than diffing screen text, which changes constantly from TUI repaints.
  5. On success it clears state, logs, and announces on the agent bus.

Verified running under launchd, not just loaded: launchctl kickstart produced agent.out/agent.err at 02:42 and rewrote .agy_quota_state at 02:42:10, with an empty error file. Log: ~/Library/Logs/agy-quota-watch/watch.log.

Aqua caveat: driving a Terminal TUI needs AppleScript, which only works from the GUI session on this host. The agent is loaded into gui/$(id -u) so it inherits Aqua. It would not work over SSH, and it cannot resume agy sessions on other hosts.

State preserved, because 3 hours is long enough to lose a Terminal

Both sessions' working context existed only in Terminal scrollback. Captured to ~/.agent-coordination/quota-hold-20260823-0242/ (transcripts + a README naming each tab's task and deadline) so the work is recoverable even if the tabs do not survive.

Peers notified on the bus not to hand these two new work until they resume.

Observation worth keeping

ttys000's subagents all report an identical elapsed time of 1h30m31s while the count crept 22 → 21. They are not progressing independently — they are almost certainly blocked on the same quota and will resume with the parent. Do not read a non-zero subagent count as "still working".