rdmpw3275m changelog — Tyrell Build 16 daemon rejected and rolled back fleet-wide

The exact signed universal Build 16 tyrelld candidate failed fleet production acceptance and was removed from service on all six Macs. Four hosts breached the 300 MB RSS acceptance ceiling, and the designated canary subsequently stopped answering both APIs while its process remained alive and accumulated 511 open-file rows. All six daemons were returned to the exact signed universal Build 15 binary at the managed stable path. Three repeated post-rollback checks found stable PIDs, 39–42 open-file rows, and HTTP 200 on both daemon ports on every host. The signed Build 16 GUI app remains installed because the observed failure is in the daemon.

Scope

• Controller and changelog host: rdmpw3275m.
• Runtime hosts changed: rdmsm4x, rdmbair15m5, rdmbair13m5, rdmpw3265m, rdmpw3275m, and jdmbair13m5.
• Daemon candidate rejected: SHA-256 69c56725e9f7708b4c7c803a7ae4639a4f470e2f6c16180730e4de780cf86eb2.
• Daemon rollback installed: SHA-256 dbc440bd54b85a54dd6c4150b23b3e508bff7d8d8080687d81b9ba7ac2cbbcf0.
• GUI app retained: Tyrell 0.2.0 Build 16, executable SHA-256 3150151bbefd1f4e76820df647208775f19709b595895b8ee0dbbf074a3be5c8.
• No Tyrell database, provider history, credential, TCC record, user data, Git history, or rollback artifact was deleted.

Acceptance failure

The unmodified 30-minute acceptance probe completed with each original PID and flat descriptor count, but four of six hosts exceeded the absolute 300 MB RSS ceiling:

The 30-minute slope metric is independently known to be too short for bursty growth (ISSUES.md #69), and the reported sessions=0 field is a known instrumentation defect (ISSUES.md #59). Neither caveat clears the four unambiguous absolute RSS failures.

Raw probe logs are retained at:

/Users/richh/.devmon/incidents/20260909-1515-tyrelld-build16-rss-acceptance-fail/

SHA-256 checks:

• jdmbair13m5.log: a94f6e3321eb4fda7eeb81578c312db55674f34add317fcd535bd541561d0667
• rdmbair13m5.log: 4a6a74b67fad05b5f73676b0f52f3ed4249c19414edd959e504c8c4fc60705b8
• rdmbair15m5.log: 9f515bc283416267ee190790b8d780a3ba45b9f289fa7e51a287558e12bf1b48
• rdmpw3265m.log: c71bcccd95aaaa43451a696111dd9e47a770dcecf1bd5902eba18d1392db48ea
• rdmpw3275m.log: 0c36f628133902aab9c739628c89ea738883863d8e3d15423b3c813822fcb58a
• rdmsm4x.log: 50c8dd9e29afb2e3aaca7d639966e93f2ce6896f7e1d8939bb72bd0688d1d3e2

Designated-canary endpoint failure

At approximately 15:26 EDT, after the nominal 30-minute probe pass, rdmbair15m5 candidate PID 40384 remained alive and launchd still reported runs = 1 and last exit = (never exited), but both local ports 43117 and 43118 stopped responding. RSS was only 243248 KB, so this was not an RSS-threshold kill.

The exact lsof capture contained 511 rows, including 433 TCP entries in CLOSED and 31 in CLOSE_WAIT, compared with 40 rows at probe completion minutes earlier. This late failure invalidates both the earlier 90-second canary receipt and the apparent 30-minute canary pass.

The complete canary capture is retained on rdmbair15m5 at:

/Users/richh/.devmon/incidents/20260909-1527-tyrelld-build16-canary-socket-leak/

Key SHA-256 checks:

• tyrelld: 69c56725e9f7708b4c7c803a7ae4639a4f470e2f6c16180730e4de780cf86eb2
• openfiles.txt: f636e8fc8fc6146a8e8f3b0131aad58c85bbcb025f19b3a23d48ec96c5b4895d
• sample.txt: 66677e3760c2b56041af19321c48273b921ec478291e32bd788794f9a02efbf0
• process.txt: c2faa444702aa63f8019501c4a8c0550b86edd60f8b3d88fb4b48e67120710d1
• launchctl.txt: 21dee43f304207d0e6f5668aa55b3ecbef6a026e31f2c6dce6e39eddf942ec2b
• vmmap-summary.txt: 0b1ba66e0b54aba440b89927177d6f7d3a80fd75c670467aaa268a76bd90e77a
• footprint.txt: 7d36fed9110c832f135b3d7bfdd12c0ea47549413599c9a9e9bde002df1718

The process sample showed multiple concurrent request tasks blocked in ECS0HTTPServer.processRawHTTP, through the Tyrell route handler, in AgentPermissionCollector.readRows(at:) at sqlite3_open_v2. Because the server closes a connection only after the async handler returns, repeated /api/agent-permissions polling can retain accepted connections while the TCC database operation blocks. This is the leading source-level hypothesis, not yet a verified fix.

Containment and rollback

• Reused the hardened stable-path installer from source implementation commit 869503aa054c9d762541c0359d07a8aefb2f2187.
• Supplied each host's already retained exact Build 15 binary to scripts/install_service.zsh --binary ..., using --server on rdmsm4x and --client on every spoke.
• The installer reverified the artifact, copied it into the retained hash-addressed release, atomically repointed the managed executable, and reloaded the per-user LaunchAgent.
• rdmpw3275m, rdmbair13m5, rdmpw3265m, and rdmsm4x rolled back at approximately 15:24 EDT.
• rdmbair15m5 rolled back after the late-hang capture at approximately 15:28 EDT.
• jdmbair13m5 rolled back through rdmsm4x at approximately 15:28 EDT.
• The local transient RSS-probe LaunchAgent com.rdm.tyrelld-build16-rss-probe had automatically relaunched once after probe exit status 1 and was booted out; it is no longer loaded.
• The Build 16 GUI app was deliberately not rolled back. Its signed universal executable remains installed and running on all six hosts.

Representative commands used for capture, artifact verification, rollback, and verification were:

for f in /Users/richh/.devmon/incidents/20260909-1515-tyrelld-build16-rss-acceptance-fail/*.log; do
  printf '%s ' "$(basename "$f" .log)"
  rg 'slope_MB_per_min=' "$f" | head -n 1
done

shasum -a 256 "$HOME/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld"
lipo -archs "$HOME/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld"
codesign --verify --strict --verbose=2 "$HOME/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld"

zsh package/scripts/install_service.zsh --client --binary "$HOME/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld"
zsh package/scripts/install_service.zsh --server --binary "$HOME/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld"

launchctl print "gui/$(id -u)/com.eastcoastscience.tyrelld"
lsof -nP -p "$(pgrep -x tyrelld)"
curl --max-time 5 -o /dev/null -w '%{http_code}\n' http://127.0.0.1:43117/api/status
curl --max-time 5 -o /dev/null -w '%{http_code}\n' http://127.0.0.1:43118/api/status

Post-rollback verification

At the final three repeated fleet probes, every daemon held its replacement PID, launchd reported it running, both local APIs returned HTTP 200, and the canonical server's /api/fleet/usage returned HTTP 200.

The rollback daemon is universal x86_64 arm64, Team Identifier ZU2882L4HT, Hardened Runtime enabled, and passes strict code-signature verification on all six hosts. This is prompt service containment only. Build 15 still has the previously documented long-duration high-RSS defect and must not be reported as a durable memory fix.

Bug and coordination records

• Filed high-severity bug ISSUE-20260909-12, related to the original lifecycle incident ISSUE-20260905-23.
• Added the acceptance and late-canary evidence to both tickets.
• Sent fleet RSS failure escalation 20260909-152228-AB9C3C1F.
• Sent corrected canary-hang and full-rollback escalation 20260909-153116-FE676AC6.
• The rejected Build 16 daemon hash must not be redeployed.

Backups and undo

• Every host retains immutable hash-addressed daemon releases and prior LaunchAgent backups created by the hardened installer.
• The rejected Build 16 daemon remains preserved only for diagnostics and reproducibility; it is not active.
• The Build 7 GUI rollback remains preserved and strict-signed on all hosts, but no GUI rollback was needed.
• Do not undo this containment by reinstalling daemon SHA-256 69c56725e9f7708b4c7c803a7ae4639a4f470e2f6c16180730e4de780cf86eb2.
• The correct forward recovery is a newly built, signed universal daemon with a new hash, installed through the hardened stable-path lifecycle after isolated regression tests, designated-canary validation, and a materially longer same-PID soak that monitors both RSS and connection state.

Outstanding work

• Replace per-request synchronous TCC permission-store collection with bounded background refresh plus a cached snapshot served by the HTTP routes, or prove a different root cause with an isolated reproduction.
• Add regression coverage for concurrent/repeated /api/agent-permissions polling and ensure blocked permission collection cannot retain unbounded connections.
• Build and sign a new universal daemon artifact on canonical rdmsm4x.
• Canary the new exact hash on rdmbair15m5, including connection-state and endpoint checks beyond the former 90-second and 30-minute windows.
• Complete a same-PID long soak across all six hosts before production acceptance.
• Integrate stable installer implementation 869503a or an independently equivalent lifecycle gate into canonical source.
• Continue monitoring the locally started agy work. All tracked parents remain locally complete with canonical accept/reject receipts pending; no tracked parent is currently eligible to resume and the user-opened interactive agy process is idle at an empty prompt with no selected project or provider turn.