rdmpw3275m-changelog-20260914-1130-tyrell-r9-lsof-rollback-r10-libproc-launch
Diagnosed R9's second lsof-induced false rollback, preserved and verified all evidence, replaced global mount-dependent probes with pinned per-PID libproc measurements, and relaunched exact Tyrell Build 20 as R10 on the designated canary only.
Scope
- Coordination and harness work:
rdmpw3275m. - Canary install/runtime and rollback validation:
rdmbair15m5only. - Canonical evidence preservation:
rdmsm4xonly. - No other fleet host received an app, daemon, configuration, or runtime change.
- No Git history, authentication store, provider session, live database schema, or pinned database backup was altered.
Files changed or created
- R10 harness:
/Users/richh/Documents/ChatGPT/rdmpw3275m/handoff/tyrell-r10-harness-20260914/canary_guard.pylibproc_probe.pypolicy.jsonlaunch_guard_power_protected.zshlibproc_layout.cprobe_lsof_variants.pyrecursive_manifest.pysummarize_samples.pytest_libproc_guard.py
- Coordinator receipt and changelog:
/Users/richh/Documents/ChatGPT/rdmpw3275m/handoff/tyrell-build20-r10-launch-20260914-1125/ - Shared project index:
/Users/richh/Library/Mobile Documents/com~apple~CloudDocs/Codex/projects/AGENTS.md principles/PROJECTS.md - Active canary stage:
/Users/richh/.tyrell/build20-canary-r10-20260914-96a5fc0onrdmbair15m5. - Canonical receipt:
/Users/richh/dev/_handoff/tyrell-build20-memory-20260912/canary-r9-lsof-r10-libproc-launch-20260914-1125/README.mdonrdmsm4x. - Complete preserved R9 stage:
/Users/richh/dev/_handoff/tyrell-build20-memory-20260912/canary-r9-rollback-20260914-1057/build20-canary-r9-20260914-96a5fc0onrdmsm4x. - Ticket
ISSUE-20260914-09was reopened; all six related Tyrell tickets received the R9/R10 receipt.
Root cause and remediation
- R9 proved that
lsof -b -w -nPstill performs work coupled to global mount state. Its exact text-path probe timed out after ten seconds while Time Machine was actively copying, even though Build 20 itself remained healthy. - Guard v1.5 removes all
lsofcalls. The pinned standard-library-only module uses per-PID macOS libproc calls to verify executable path, descriptor count, and TCP count. - Descriptor churn retries a complete snapshot at most three times; buffer and structure sizes are bounded and validated. Identity, APIs, descriptor delta, TCP count, power, sample continuity, database, and automatic rollback remain fail-closed.
Commands and verification
- Ran host/agent preflight, ticket ownership checks, and fleet mailbox synchronization.
- Preserved R9 by direct canonical rsync, then compared deterministic recursive manifests by relative path, mode, size, content hash, directory, and symlink identity.
- Verified R9 exact Build 15 daemon and Build 17 app rollback, launchd state, both status APIs, database quick-check, file count/additive column, app/bar counts, and absence of the R9 guard/assertion.
- Compiled the native libproc layout probe on Intel and Apple Silicon; both architectures reported the same required sizes and offsets.
- Ran 30 consecutive full guard measurements during active Time Machine copying: maximum duration 0.061 seconds, exact path, descriptor count 25, TCP count 2, and zero lsof calls.
- Ran injected no-lsof and descriptor-churn contracts, a missing-module fail-before-mutation test, Python/JSON/zsh validation, exact dependency validation, and a complete disposable rollback transaction.
- Prepared a fresh R10 stage from the post-test exact Build 15/17 baseline, ran the pinned app dry-run, promoted exact notarized Build 20, completed launch preflight, and started only through the pinned power-protected launcher.
- Verified R10 guard/caffeinate parentage, both sleep assertions, AC/open envelope, exact app and daemon hashes, launchd PID/start/path/runs, both status and permissions planes, database integrity, and two immutable initial samples.
Evidence
- R9 source/canonical manifest SHA-256:
3156eb60423fc8c8a91f7fb3d3e54b82d6957931572f6fede942cd9599adcfd6; 53 files, 30 directories, zero symlinks, 275,051,968 bytes. - Guard v1.5 SHA-256:
76fd2739188affe2190d997338bc0a6c3349c1b0649861372b25a7bc53dfd1be. - libproc probe SHA-256:
c960d7b16d55c616269c1c3f8b98dab1642906cafade0a26aa08311af3cff9cc. - Policy SHA-256:
3b2e2d245427176787d32f173242b1a1fcd4eb0bfb4564c047b51e11da5ead6d. - Launcher SHA-256:
53f0236f0c9d5e5c253da11d7f0d7cc8235771ccd39c60c521152a74c218f81e. - Canonical receipt SHA-256:
312464125622d9dba62ac22e55db7f69683e3cb01434b909ca83e552898477f8. - R10 exact daemon/app SHA-256:
71bf6122e453315205f3f0dfaf3f1a3a6cf2d99dfe901cda88e7ac79a8112c77/423871cede8ae6313c486929f544dbee8d1085540b2d37896de6f2c2d4de90fd. - R10 daemon PID/start:
4857,Mon Sep 14 11:25:27 2026; guard/caffeinate:4570/4692; app/bar:4112/4145.
Recovery and undo
- R10 retains a complete exact Build 15 daemon, plist, Build 17 app, and pinned database-backup verification payload before any service mutation.
- The guard automatically restores the exact signed Build 15/17 baseline on any failure.
- A bounded manual rollback, if independently required, is
python3 /Users/richh/.tyrell/build20-canary-r10-20260914-96a5fc0/canary_guard.py rollbackfrom the canary's local GUI/owner context. - Never restore the pinned pre-migration database over the additive live database.
- Complete R8 and R9 stages and app rollback bundles remain retained for diagnosis and recovery.
Outstanding work
- R10 is
SOAKING_NOT_ACCEPTED; warmup, short, six-hour, fixed 24-hour, second unchanged inventory, CPU, fresh app, and controlled-reboot gates remain pending. - No fleet expansion is authorized.
ISSUE-20260914-09remains in progress until R10 proves the replacement beyond the prior failure interval and required gates.- Track B agy has no eligible unfinished parent or real quota-reset event; canonical handoff acceptance remains pending.