herdr fleet rollout — install, service, integrations, single-sidebar attach

Session: claude@rdmbair15m5 · Started 2026-09-05 04:31:50 EDT · Ended 2026-09-05 05:10 EDT (~38 min) Goal: install herdr across the Mac fleet, run it as a service by default, detect/integrate the agents, attach every host into one sidebar, name each machine, add a footlights-style status bar, and authorize desktop control.

Result

Integrations installed per host: claude, codex, grok, antigravity-cli (+cursor on rdmsm4x). permissions.defaultMode = bypassPermissions verified intact on all six after the Claude hook landed.

Artifacts (canonical: rdmsm4x:~/dev/fleet/herdr/)

• fleet_herdr_setup.zsh v1.0.0 — install, config, service, integrations, verify (--dry-run)
• fleet_herdr_attach.zsh v1.0.0 — one workspace per host in one sidebar (--dry-run, --close)
• herdr_tcc_grant.zsh v1.0.0 — --register (safe, unattended) / --request (human at keyboard)
• config.toml — the fleet herdr config
• ISSUES.md — 4 OPEN, 3 RESOLVED
• fleet_herdr_label.zsh v1.0.0 — refresh sidebar tokens (project, branch, agy sub-agent count)

Config sha 2af2f63fdfb6 verified identical on 6/6 at 05:14 EDT, reload-config = applied on all. Sub-agents are surfaced as a COUNT token ($subagents), not as panes: 113 on rdmbair15m5, 80 on rdmpw3275m, 60 on rdmsm4x, 0 elsewhere. fleet_herdr_label.zsh needed ssh -n — without it ssh swallows the loop's stdin and only the first host or two get labelled, silently.

Config: footlights-style bar and per-machine naming

ui.tab_bar_right = zoom · hostname · datetime %Y-%m-%d %H:%M:%S · command date +%Z (hourly). These entries resolve on the SERVER, so a remote attach names the host the panes actually run on. %Z is NOT a supported datetime component in 0.8.2 (diagnostic: "component is not supported at byte index 19"), hence the hourly command entry supplying the zone — Rich's timestamp rule requires a named zone. Also status_indicators = "symbols", window_title = "{hostname}: {workspace}", and sidebar rows [["state_icon","workspace"],["$project","branch","git_status"]] where $project is a custom token fed by herdr workspace report-metadata --token project=<dir>. Sidebar now reads claude@<host> with project=dev for all six.

Three traps found and recorded

1. herdr server starts happily on a BROKEN config, logging only a WARN and falling back to defaults. "It started" validates nothing. The real validator is herdr server reload-config, which returns .result.status = applied|partial|failed with diagnostics — but exits rc=0 even when it fails, so the status must be parsed in-band. Found via a deliberate negative control.
2. herdr integration status never prints "installed" — it prints current (vN). A verify step grepping ': installed' counted 0 on every host while all integrations were in fact fine.
3. TCC: making herdr a service is what removes its privileges (HERDR-20260905-04).

Cross-host integration: what works, what doesn't

Verified working: remote-forwarding the local herdr API socket (ssh -R) and exporting HERDR_ENV=1 / HERDR_SOCKET_PATH / HERDR_PANE_ID on the remote makes a remote agent's herdr hook report into THIS Mac's server. Proof: herdr pane list run on rdmsm4x through the forwarded socket returned rdmbair15m5's panes byte-identically, and remote Claude sessions reported their agent_session IDs into local panes. Verified NOT possible in 0.8.2: live remote agent STATE. herdr claims a pane by local process detection; over ssh the foreground process is ssh, and pane report-agent will not create occupancy. See HERDR-20260905-03.

Not done / handed off

• Screen Recording, Accessibility and Full Disk Access for the launchd herdr still need one human click per host, or an MDM PPPC profile. Placeholder rows created (flags=0, i.e. clickable).
• claude is 20–146s to start on rdmbair13m5 / jdmbair13m5 — pre-existing, not herdr (HERDR-20260905-06). Suggest macos-log-triage on both.
• codex is at 100% of its OpenAI weekly quota (resets 2026-09-07); grok is installed-not-authenticated fleet-wide. Neither blocks herdr.

No secrets were written to any file, note or log.

Resume session — 2026-09-05 11:44–11:55 EDT (claude@rdmbair15m5)

Between sessions the fleet workspaces had been recreated as shell@<host> (an attach re-run with --start none), so the sidebar held signed-in shells rather than agent sessions. Restored:

• All 5 remote panes confirmed genuinely signed in (foreground process ssh, remote starship prompt showing [ rdmsm4x ], [ rdmpw3275m ], ...). Sign-in survived the gap.
• Launched Claude into the 3 healthy hosts; relabelled by ACTUAL pane content, not assumption: claude@rdmsm4x, claude@rdmpw3265m, claude@rdmpw3275m; shell@rdmbair13m5, shell@jdmbair13m5 stay shells because Claude cannot start there.
• Service state re-verified: 5/6 started, rdmbair15m5 still error for the documented socket-ownership reason (self-heals at next login).
• Sub-agent counts refreshed: 127 rdmbair15m5 · 80 rdmpw3275m · 69 rdmsm4x · 0 elsewhere.

Two of my own bugs found and fixed while resuming, both silent-failure shaped:

• zsh does not word-split unquoted parameters. set -- $pair inside a loop put "w9:p1 rdmsm4x" entirely into $1, so herdr pane send-text got an invalid pane id and no-opped with rc ignored. The tell was an empty () in my own echo output.
• fleet_herdr_label.zsh mis-parsed duplicate labels. herdr appends #2/#3 to duplicate workspace labels, so the host parsed as rdmbair15m5#2 and reported subagents=0. Now stripped.

HERDR-20260905-06 (claude hangs on both 13m5 Macs) updated with FOUR disproved hypotheses - orphaned daemons, Gatekeeper quarantine, binary read/IO, and Claude state size - each tested and killed, so the next person does not repeat them. Still open.

12:16–12:25 EDT — approved-items pass, and a CRITICAL unrelated incident

Rich approved the three blocked items ("all approved and run with it", "trust my home dir", "I'll fix the TCC shortly"). Outcome:

• Full session resume — moot. wG:p1 had already resumed itself and finished at 12:08 EDT; no prompt remained, so the 544k-token spend never happened.
• Home-dir trust — no prompt pending; a fleet-wide resume had already cleared them. Swept all six hosts: zero agents in blocked state.
• TCC — left to Rich at his request.

HERDR-20260905-09 (CRITICAL): gopls-lsp plugin exhausted the system file-descriptor table. rdmbair15m5 reached kern.num_files 2,097,093 / 2,097,152 (99.997%); every command failed with ENFILE: file table overflow and even /dev/null could not be opened. lsof attributed 2,085,254 descriptors to gopls — 99.4% of the whole system limit — spawned as go run golang.org/x/tools/gopls@latest mcp by the gopls-lsp plugin, once per session, accumulating for 11h+. Killed: rdmbair15m5 2,097,093 -> 11,832; rdmpw3275m 3,381,198 -> 11,557. The plugin itself was NOT disabled — that is Rich's setup decision. Raising kern.maxfiles is not a fix: it is already at 2M/5M on these hosts, far above the macOS default, and the leak ate it.

HERDR-20260905-06 refined: the 13m5 slowness is STARTUP-ONLY. Both hosts now run 4 live Claude agents each; claude --version still takes 90s. Five hypotheses now disproved, the fifth being fd exhaustion (those hosts measured 0% and 3%).

Fleet health at 12:23 EDT: herdr 0.8.2 6/6 · service started 5/6 · integrations 4–5/host · fd usage 0–3% on all six · gopls 0 on all six · 27 native agent sessions fleet-wide. Transient: rdmsm4x refused ssh at kex (MaxStartups throttling, partly self-inflicted by the mesh plus repeated probes); the ISSUES.md push was queued and landed at 12:23:04 with sha parity 38923351f467a27b / 191 lines.