Fleet changelogs · dev.ecs0.net
rdmpw3275m-changelog-20260912-0038-tyrell-build19-six-hour-rejection-rollback

Tyrell Build 19 failed the real six-hour charged-physical-footprint gate on the designated canary; the fail-closed guard restored the exact signed Build 15 daemon and Build 17 app, and a new canonical bug now owns the allocator remediation.

Tyrell Build 19 six-hour rejection and verified rollback

Outcome

The exact signed Build 19 canary was rejected at its first long-runtime checkpoint. This was not the earlier RSS-metric false positive: the corrected physical-footprint-v2 gate measured charged physical footprint and found both the endpoint and growth rate outside policy. The guard then performed the expected rollback without operator intervention.

No fleet rollout occurred. Rejected Builds 16, 18, and 19 remain prohibited. The designated canary is back on the exact signed universal Build 15 daemon and retained Build 17 app while a narrower source remediation is developed.

Exact Build 19 identity and continuity

Six-hour gate result

The post-warmup window ran from 2026-09-11T18:28:20.629348-04:00 through 2026-09-12T00:28:23.378465-04:00, spanning 21602.749116897583 seconds.

Inventory pass 7 completed at 00:25:32 EDT, approximately three minutes before the fixed gate endpoint. The guard raised ValueError: physical footprint exceeds 300MiB bound; no long-gate receipt was written because validation failed before receipt creation.

Failure allocation evidence

The failure-time footprint capture measured 669058896 charged bytes. It was allocator-dominated:

This evidence supersedes any explanation that treats the failure as reclaimable RSS alone. The gate semantics remained unchanged; the underlying inventory/allocation behavior now requires correction.

Rollback verification

The guard wrote state rolled_back with reason physical footprint exceeds 300MiB bound and restored:

Build 15 predates the cached permission endpoint, so /api/agent-permissions returning 404 after rollback is expected and was not treated as a rollback failure. The two service status endpoints are the compatible rollback health check.

The rejected Build 19 app was retained recoverably at /Applications/.tyrell-rollbacks/20260912-002826-933b995d2964/Tyrell.app. The rejected daemon and complete evidence remain under the canary stage; neither was deleted.

Evidence locations and hashes

Evidence root on rdmbair15m5:

/Users/richh/.tyrell/build19-canary-20260911-ec03697

Canonical records changed

Dispatch is not receipt, source acceptance, integration, artifact creation, or release. Those states remain pending.

Representative verification commands

ssh [email protected] 'jq -c . /Users/richh/.tyrell/build19-canary-20260911-ec03697/state.json'
ssh [email protected] 'tail -18 /Users/richh/.tyrell/build19-canary-20260911-ec03697/samples.jsonl'
ssh [email protected] 'tail -12 /Users/richh/.tyrell/build19-canary-20260911-ec03697/inventory.log'
ssh [email protected] 'cat /Users/richh/.tyrell/build19-canary-20260911-ec03697/failure-footprint.txt'
ssh [email protected] 'launchctl print gui/$(id -u)/com.eastcoastscience.tyrelld'
ssh [email protected] 'shasum -a 256 "/Users/richh/Library/Application Support/Tyrell/bin/tyrelld" "/Applications/Tyrell.app/Contents/MacOS/Tyrell"'
ssh [email protected] 'curl -fsS http://127.0.0.1:43117/api/status; curl -fsS http://127.0.0.1:43118/api/status'
/Users/richh/bin/ticket show ISSUE-20260912-01
AGENT_LLM=codex /Users/richh/.agent-coordination/agent_msg.zsh read 20260912-003702-10A4C8FE

Recovery and undo

The current rollback is the safe state and should not be undone while ISSUE-20260912-01 is open. If controlled reproduction is required, use the retained canary artifacts and the pinned lifecycle scripts under a new rollback-protected canary; do not point launchd at a mutable build tree and do not reinstall rejected Build 19 as production.

To recover from any later candidate failure, reinstall the exact retained Build 15 daemon through the hash-addressed stable-path lifecycle and reinstall the retained Build 17 app through the signed bundle lifecycle, then verify exact hashes, signature, launchd state, both status endpoints, process identity, descriptors, and sockets.

Outstanding work

  1. The canonical owner must acknowledge and accept ISSUE-20260912-01 after collision preflight.
  2. Diagnose the allocator-heavy hourly inventory path without weakening physical-footprint-v2.
  3. Add deterministic allocation/regression coverage and pass the full canonical Swift 6.4 suite.
  4. Independently review the source correction.
  5. Build, sign, and notarize a newer universal Tyrell app and daemon with exact hashes.
  6. Run another designated rollback-protected canary whose inventory pass overlaps the six-hour endpoint, then complete the existing 24-hour memory, CPU, descriptor/socket, functional-app, stable-path reboot, signature, rollback, and exact-hash gates.
  7. Do not expand to the fleet until all acceptance evidence exists.

The separate local agy monitor was also checked. No agy process is running, every discovered parent remains complete or awaiting canonical acceptance, and no provider quota-reset event or eligible incomplete parent permits a resume.