rdmsm4x — nightly dev_update sweep "exit 145": tyrelld FleetGuard SIGSTOP, fixed in Tyrell c76343f

Summary: For two nights the fleet sweep read "exit 145" on 4 of 6 hosts and skipped their real updates. The cause is Tyrell's daemon pausing (SIGSTOP) any process whose arguments mention dev_update. The fix is merged to Tyrell main (c76343f) with tests; the signed rollout is handed to the Tyrell lane.

• Host: rdmsm4x (the fault is fleet-wide: tyrelld runs on all six)
• Agent: claude@rdmsm4x (Claude Code, session 4fec0f54)
• Tickets: ISSUE-20260927-20 (handed to tyrell@rdmsm4x for deploy) · ISSUE-20260927-22 (unrelated fd-limit test)
• Window: 2026-09-27 17:40 → 17:51 EDT

Evidence

• fleet/overnight/dev-update-2026092{6,7}-0307.md:
  • 09-25 was clean on v2.14.
  • 09-26 got 145 on 3 hosts, still on v2.14, so this is not a script change.
  • 09-27 got 145 on 4 hosts.
• rdmpw3275m ~/.local/state/fleet_update/last_run.json: exit_code: 10, yet the sweep saw 145. The status changed after the script's verdict.
• 145 = 128 + 17 = SIGSTOP on macOS; zsh reports a stopped child that way.
• Stopped processes were live on rdmsm4x: a tee -a ~/scripts/dev_update_20260927_172833.log (stopped 15+ min, past the 120 s TTL) and an awk.
• sudo eslogger signal for 120 s: tyrelld (pid 38838, com.eastcoastscience.tyrelld, team ZU2882L4HT) sent signal 17 to /usr/bin/awk.
• Source: Tyrell/Sources/TyrellCore/Guards/FleetLoadPolicy.swift, isDeferrableProcess: cleanArgs.contains("dev_update") → ProcessHoldManager.holdProcess → kill(pid, SIGSTOP), TTL 120 s.

Fix (Tyrell main c76343f, fast-forward from 85b5dda, pushed to private backup)

• dev_update / fleet_update count as shared-lock holders and are never held. They hold Homebrew's lock and the dev_update run lock, and their caller waits on the exit status. This is the same class as ISSUE-20260927-06 (ReplicantDB, gitmirror).
• The remaining deferrable jobs (devbackup, bottlemirror, fleet probe) now match on the script being run: argv[0], or the first non-option argument after an interpreter. They no longer match any substring of args.
• Tests:
  • FleetGuardCoordinatorTests 6/6.
  • Negative control: the same tests against 85b5dda fail 7 assertions.
  • TyrellCoreTests: 1018 swift-testing tests passed, plus the XCTest suites. The only failure is FileDescriptorLimitTests.testRaiseNeverLowersTheLimit, which fails identically on 85b5dda (host soft fd limit 1048576 > kern.maxfilesperproc 245760), filed as ISSUE-20260927-22.
• Worktree: ~/dev/_worktrees/tyrell-issue-20260927-20, branch fix/fleetguard-no-hold-dev-update-20260927.

Remaining (owned by the Tyrell lane, bus + ticket handoff 17:50)

• Signed Tyrell build containing c76343f, rolled out through the canary/receipt pipeline, ideally before the 03:07 sweep.
• Then the next sweep report should show no 145.
• Lane observation: a held pid re-held every cycle stays stopped far past its TTL.

Undo

git -C ~/dev/apps/Tyrell revert c76343f. Nothing was deployed from this session. Runtime state touched: none, except that the orphaned tee had already exited when a SIGCONT was attempted.