Fleet changelogs · dev.ecs0.net
rdmpw3275m-changelog-20260910-1348-tyrell-build17-daemon-rejected-build18-fix

Tyrell Build 17 daemon rejected; six-host containment restored; Build 18 fix published

Tyrell Build 17 remains installed as the signed and notarized GUI app, but its separately rebuilt tyrelld was rejected. The rollout had not integrated either the stable executable lifecycle or the TCC permission-cache correction, so all six hosts again executed non-identical daemons from mutable SwiftPM build output. All six daemons were returned to the exact signed universal Build 15 rollback at the managed stable path, and a Build 18 source correction was validated and published for canonical review.

Scope

Why Build 17 daemon was rejected

Canonical main 53c0d8c still contained five synchronous AgentPermissionCollector.collect calls in the daemon request/push/poll paths. The prior Build 16 canary capture had directly sampled ten data-plane handlers plus the permission-push loop blocked in AgentPermissionCollector.readRows(at:) through sqlite3_open_v2. Because the HTTP server closes a connection only after the awaited handler returns, repeated polling retained accepted sockets and eventually wedged both status planes while the process remained alive.

Canonical main also still contained install_service.zsh v1.0, which renders the LaunchAgent against .build/release/tyrelld. The Build 17 deployment therefore rebuilt and launched a different daemon on each architecture/host instead of promoting one exact signed artifact to the managed path. Fresh post-reboot evidence was:

Host Pre-containment executable SHA-256 RSS before rollback
rdmsm4x .build/out/Products/Release/tyrelld 7a6b343d7bf98782d0de97f17d62e660397314700eb9abb99abc343d372ddcc3 2,826,512 KB
rdmbair15m5 .build/out/Products/Release/tyrelld f908abf9ed42e6eda48360237ca541c098b6cf115c07d893d598ea75ca16d4b0 596,096 KB
rdmbair13m5 .build/out/Products/Release/tyrelld 14edfe705f78d0276ded9f88b769c1c30de53dc03b460083f508b982b2190789 481,712 KB
rdmpw3265m .build/x86_64-apple-macosx/release/tyrelld 5464635b13a415c89ec67369556b62691464eb1ab031305115f11fae7d926fe8 496,920 KB
rdmpw3275m .build/x86_64-apple-macosx/release/tyrelld b9ff7a787567069a10955b53478fb08a28a2cca69ee31ff85fcac67cc8a1b154 1,576,412 KB
jdmbair13m5 .build/out/Products/Release/tyrelld 14edfe705f78d0276ded9f88b769c1c30de53dc03b460083f508b982b2190789 299,168 KB

This contradicts the Build 17 release checkpoint's claim that nothing was outstanding. The notarized Build 17 GUI bundle is not the failed component and remains installed.

Runtime containment

Each host already retained the same rollback executable at:

~/Library/Application Support/Tyrell/releases/daemon-dbc440bd54b8/tyrelld

Before use, every reachable copy was independently measured as:

The hardened installer from the source branch was copied additively under each host's incident directory and invoked with --server only on rdmsm4x; all spokes used --client. It retained the release, atomically repointed ~/Library/Application Support/Tyrell/bin/tyrelld, rendered the LaunchAgent against that stable path, reloaded the per-user job, and waited for status.

rdmpw3265m and rdmpw3275m had rebooted with Tailscale BackendState=Stopped. Their existing logged-in connections were started with tailscale up; no account, node identity, credential, DNS, or ACL was changed. The former was reached over its observed LAN address only long enough to start its existing tailnet connection, after which all work used the pinned tailnet name.

Three repeated post-rollback probes on every host observed:

Host PID Role Daemon SHA-256 Open-file rows 43117 43118 GUI build
rdmsm4x 10993 server dbc440bd… 40–42 200 200 17
rdmbair15m5 49512 client dbc440bd… 40 200 200 17
rdmbair13m5 28419 client dbc440bd… 41 200 200 17
rdmpw3265m 18895 client dbc440bd… 39–41 200 200 17
rdmpw3275m 48734 client dbc440bd… 40–41 200 200 17
jdmbair13m5 20228 client dbc440bd… 39 200 200 17

Build 15 is prompt containment only. Its separately documented long-duration RSS defect remains, and devmon restart protection must stay enabled until the new exact artifact passes a long soak.

Build 18 source changes

The branch contains three commits on top of current canonical main:

  1. 2a86280 — isolate TCC collection behind an immutable encoded snapshot cache and one single-flight native refresh queue. HTTP routes, the client push loop, and the fleet poller only consume the last completed snapshot. Before the first snapshot, the permission endpoint returns HTTP 503 rather than blocking. Cached replies report age from the collector's observedAt.
  2. 3691fdf — transplant the verified stable daemon lifecycle from original implementation 869503a: require one signed universal Team ZU2882L4HT Hardened Runtime candidate, retain it by SHA-256, and launch only the managed stable path.
  3. c0a233e — advance the candidate stamp from Build 17 to Build 18 and add the evidence-bounded SESSION-STATE.md / ISSUES.md checkpoint.

Exact project files changed by the published branch:

Commands and verification

zsh Tests/ScriptTests/agent_permission_cache_contract_test.zsh
# [PASS] tyrelld serves cached agent-permission snapshots; no daemon route or poller performs direct TCC collection

zsh Tests/ScriptTests/tyrelld_stable_install_contract_test.zsh
# [PASS] tyrelld install is independent of mutable build output

zsh Tests/ScriptTests/version_single_source_contract_test.zsh
# [PASS] one source of truth: VERSION=0.2.0 BUILD_NUMBER=18 reach both Info.plists; drift and second copies fail closed

swift build --product tyrelld --jobs 2
# Build of product 'tyrelld' complete

swift build --target TyrellCoreTests --jobs 2
# Build of target: 'TyrellCoreTests' complete

swift run --package-path \
  /Users/richh/.devmon/incidents/20260909-tyrelld-permission-cache-harness Harness
# PASS blocked_collector_calls=1 cached_reads=100000 duplicate_refresh=false final_host=fresh

git diff --check
git push -u origin codex/tyrelld-permission-cache-20260909
git ls-remote --heads origin codex/tyrelld-permission-cache-20260909
# c0a233e6d5d7ab450fded7516ce9c5af1fae517c

The isolated daemon stress record is:

/Users/richh/.devmon/incidents/20260910-tyrelld-permission-cache-stress.ypxiaI

Its raw result was:

initial_fd=40 final_fd=41 permission=200 data_status=200 control_status=200 requests=1000 successes=1000 codes=1000 200

The full command swift test --filter FleetAgentPermissionTests --jobs 2 is not a completed gate on this Intel host. Apple Swift 6.3.3 crashes while compiling the unrelated Sources/tyrell-app/SettingsView.swift IRGen with signal 6 and SmallVector unable to grow; Requested capacity (4294967297). The new test target compiles. This is recorded as a toolchain/ target-graph blocker, not misreported as either a permission-test failure or full-suite success.

Evidence and backups

Undo and forward path

Do not undo containment by reinstalling Build 16 daemon 69c56725... or any of the six per-host Build 17 daemon hashes. If a diagnostic rollback of this containment is explicitly required, the prior plist is in each host's incident directory and the former executable still exists in that host's .build tree, but doing so would knowingly restore both open incidents.

The correct forward path is canonical integration of published head c0a233e, an exact signed universal Build 18 daemon plus notarized app built in the authorized rdmsm4x GUI lane, and an exact-hash rdmbair15m5 canary. That canary must include concurrent permission requests, descriptor/socket-state checks, same-PID RSS measurement beyond the former 30-minute window, retained rollback verification, and a reboot check proving the managed LaunchAgent path survives. Only the accepted exact daemon hash may expand to the fleet.

Outstanding owner actions