Fleet sweep 1.2 — Intel real runs get room to build; every capped run leaves evidence
When: 2026-09-22 11:23:02 → 2026-09-22 11:36:03 EDT
(both read from the clock) · Host: rdmsm4x ·
Scope: the 03:07 fleet sweep (runs on rdmsm4x, reaches
all six) Ticket: ISSUE-20260922-03 (resolved) ·
Commits (~/dev/scripts): 9a89c3d (sweep
1.2), 6d584a2 (session state)
One line: last night was the first unattended run of the fixed gate — all six hosts passed it and four completed real updates — and the two that hit the time cap did so for opposite reasons, one of which the sweep had no way to explain. Both are now handled.
What last night showed (2026-09-22 03:07)
- 6 of 6 dry runs passed the gate; 4 of 6 real runs completed (rdmbair13m5, which had hung the day before, among them). No run touched the Xcode selection.
- rdmpw3275m hit the 1800s cap while working: Tier-3 Intel has no bottle for atuin, so Homebrew built it from source with cargo. The payload itself budgets 14,400s for exactly this. Worse, a remote run does not finish after the cap — it dies at its next write to the closed ssh session (atuin was left with two kegs and no run summary) — so a too-short cap would kill the same build every night, for ever.
- rdmsm4x hit the cap stuck: 29
minutes in "Fetching downloads for: atuin", with the bottle already
cached since the evening before. Not reproducible by day (2s in the
foreground, 2s under
taskpolicy -b, no sleep events, no sandbox denials), and nothing had recorded what it was waiting on.
What changed — fleet_dev_update.zsh 1.2
- Intel hosts run last, with a real-run cap of
14,400s (the payload's own Intel budget); arm64 keeps
1800s.
host_archasksuname -m; an unreachable host keeps the conservative cap. - 20 seconds before any cap — dry or real, local or
remote — the host's pinned dev_update chain, its brew/cargo/rustc/curl
processes and a 3-second
sampleof each brew process go to the end of that host's log, and the report row points there. The run is still stopped by the sametimeout, which kills the whole process group. FDU_HOSTS/FDU_*_CAP/FDU_OUToverride the defaults for tests only;FDU_OUTkeeps test reports out offleet/overnight. Report headers no longer print a literal\n.
Other local changes on rdmsm4x
- atuin 18.22.0 → 18.23.0 — the bounded reproduction of the stalled upgrade (completed in 2s).
- An accidental
brew upgrade --formularan at about 11:27 when a ticket body was written through an unquoted heredoc that contained backticks. Checked: a no-op — no Cellar keg changed, nothing was outdated. Recorded on ISSUE-20260922-03 and as a memory (unquoted-heredoc-executes-backticks).
Found, not mine, verified compatible
Another lane shipped dev_update v2.02 overnight
(commit 817bda4, on all six hosts): a keychain search-list
fix (ISSUE-20260921-24). It is a clean superset of 2.01 — same payload
sha f87802488dca, every 2.01 change present.
Verification
| test | result |
|---|---|
| capped remote dry run (jdmbair13m5, rdmbair15m5) | snapshot of the remote dev_update chain + live brew |
| capped local dry run (rdmsm4x) | snapshot, then killed; nothing left alive, no lock left |
| normal dry run | no snapshot kept |
host_arch |
arm64 / x86_64 / unknown, correct |
| real run through 1.2 (jdmbair13m5) | dry 20 → real 20, arm64, cap 1800s, 51s |
zsh wait after polling |
returns 0 / 20 / 30 / 124 under
set -uo pipefail |
Still assumed: an Intel real run finishing inside the 14,400s cap — tonight's 03:07 is the first chance.
Undo
git -C ~/dev/scripts revert 9a89c3d restores sweep
1.1.