rdmsm4x-changelog-20260904-0700-tyrell-fd-and-memory-leaks-fleet-console-login-blocker

rdmsm4x changelog — 2026-09-04 07:00 EDT

Host: rdmsm4x · Agent: claude@rdmsm4x (Opus 5, max reasoning), session 6f528866-644b-406f-a0bb-36a282e54191 · Window: 2026-09-04 06:12 → 07:00 EDT Authority: Rich, in-session 06:21 EDT — "your show, good luck and good night. you have full auth/approval to get the fleet healthy, and get us building again!"

One-line summary

Found and fixed two independent leaks in tyrelld that together took the six-Mac fleet down overnight — a 2-descriptor-per-call pipe leak and a 48.7 GB autorelease-pool leak — and established that three of six hosts are dark because nobody logged in at the console after the remediation reboots, which is also why the signed fleet deployment cannot be completed without Rich.

Scope

Code changed: ~/dev/apps/Tyrell only. 4 commits on canonical main, pushed to fleet (git.ecs0.net) and backup (GitHub).
Runtime changed: rdmsm4x only — tyrelld restarted on the fixed binary; a stopgap supervisor started. No other host was mutated in any way.
Read-only elsewhere: all five remote hosts were measured, none changed.

Files touched

File	Change
`Sources/TyrellCore/Safety/SubprocessRunner.swift`	NEW — the single Process+Pipe implementation
`Sources/TyrellCore/Safety/FileDescriptorLimit.swift`	NEW — raises RLIMIT_NOFILE at startup
`Sources/TyrellCore/ProjectIdentity.swift`	`git()` routed through the runner; git env hardened
`Sources/TyrellCore/RepoBundler.swift`	`spool()` routed through the runner; `--quiet` added
`Sources/TyrellCore/Safety/ProcessSafetyActor.swift`	`execute()` routed through the runner
`Sources/TyrellCore/Hashing.swift`	autorelease pools in `fullHash`/`partialHash`
`Sources/TyrellCore/Scanner.swift`	autorelease pools in both `HashFiller` paths
`Sources/tyrelld/ClientLoop.swift`	autorelease pool around the `onBatch` body
`Sources/tyrelld/Daemon.swift`	one line: `FileDescriptorLimit.raise()` (partial-file commit)
`Sources/tyrell/Console/TicketCLIExecutor.swift`	routed through the runner
`Sources/tyrell/Console/ConsoleRemoteClient.swift`	routed through the runner
`Sources/tyrell/Tyrell.swift`	iperf3 path routed through the runner
`Tests/TyrellCoreTests/SubprocessRunnerFDLeakTests.swift`	NEW — 9 count-asserting tests
`Tests/TyrellCoreTests/FileDescriptorLimitTests.swift`	NEW — 5 tests

Outside the repo, on rdmsm4x only:

Path	Change
`~/.agent-coordination/UNATTENDED`	CREATED on Rich's explicit in-session instruction; records who authorized it
`~/dev/_ops/incidents/20260904-.../stopgap/hub_agent_stopgap.zsh`	NEW — session stopgap for the dead LaunchAgents
`~/dev/_ops/incidents/20260904-.../MORNING-START-HERE.md`	NEW — handoff
`~/dev/_ops/incidents/20260904-.../findings/RCA-tyrelld-fd-leak.md`	NEW — root-cause analysis
`~/dev/apps/Tyrell/SESSION-STATE.md`	new checkpoint prepended; all 15 prior checkpoints preserved

Verification evidence

Claim	Evidence
FD leak, pre-fix	200 git calls: 6 → 406 descriptors (+2.0/call), 200/200 returning correct output
FD leak, post-fix	same 200 calls: 406 → 406 (0.0/call), 200/200 returning correct output
Caught live	daemon killed at 06:40 held 2,714 FDs / 2,674 pipes, up from 39/0 at 06:12
Memory leak identified	`heap -s`: 566,584 nodes / 48,687,383,808 bytes / avg 85,931 / NSConcreteData
Memory, pre-fix	3,000 real files → +3,930.9 MB (1.31 MB/file)
Memory, post-fix	same 3,000 files → +1.2 MB (0.0004 MB/file)
FD limit raise	simulated 256 baseline → 65,536 (hard=unlimited, kern.maxfilesperproc=2,500,000)
Universal2	`lipo -archs` on all 5 built products → `x86_64 arm64`, minos 15.0
Fix present in shipped binary	`nm -a`: 196 SubprocessRunner, 72 FileDescriptorLimit, objc_autoreleasePoolPush
mem0 not implicated	`nm -a .build/release/tyrelld \| grep -ci mem0` → 0
Hub has no GUI session	`/dev/console` owned by root; positive control returns 11/12 on logged-in hosts
Signing blocked	`codesign` → errSecInternalComponent; keychain "User interaction is not allowed"
Tests	14 new; full suite 523 XCTest + 396 swift-testing passing, 2 failing (both proven environmental)

Backups / how to undo

Pre-edit copies of all six originally-touched sources: /private/tmp/claude-501/-Users-richh-dev/6f528866-.../scratchpad/snap-20260904-061812/ (session-scoped — the durable copy is git history: parent commit 048c913).
Sources/tyrelld/Daemon.swift working revision before the partial-file commit: .../scratchpad/snap-daemon-064650/Daemon.swift.working.
Revert the code with git revert e828ae0 7193c00 a527972 8aeef22 in ~/dev/apps/Tyrell.
Stop the stopgap with pkill -f hub_agent_stopgap. It changes no configuration.
Delete ~/.agent-coordination/UNATTENDED to restore the attended escalation gate.

Outstanding owner actions

Console login on rdmsm4x, rdmpw3265m, rdmpw3275m — all three sit at the login window with zero com.eastcoastscience LaunchAgents running. This is the fleet outage.
Sign and deploy Tyrell from Terminal.app on the console — codesign cannot run over SSH. The adhoc build was deliberately NOT deployed: it would change the designated requirement and destroy the daemon's TCC grants.
Apple Notes entry is PENDING — notes_changelog.zsh refuses from a background session (launchctl managername = Background, not Aqua), and it is correct to. This file is the archive; the Notes entry must be added when a GUI session exists.
Consider raising launchctl limit maxfiles on the five spokes (currently 256 vs the hub's 65,536).

No secrets

No credential, token or key value appears in this file, in any commit, or in any artifact produced by this session. ~/.tyrell/token was referenced by name only.