rdmsm4x-changelog-20260825-0647-tyrell-overnight-incident-fixes

Summary: Overnight autonomous shift (03:00–06:47 EDT) on tyrell.app: two watchdog-caught fleet-drain outages root-caused and fixed live — six engine/deploy fixes, fleet redeployed three times, drain advanced 11k → 59k CAS blobs; ended paused only by a macOS background indexing storm (self-resolving).

Scope: rdmsm4x (master) + all 5 client Macs (tyrelld service redeploys). Repo: ~/dev/apps/Tyrell, branch main, commits 8866cf2..663685d (nothing pushed).

Fixes (each with regression cover where testable; 46/46 suite green throughout)

1. @4be046e 15s data-plane timeout killed 65–232s manifest handling → clients re-sent full manifests forever. 120s cap + public timing telemetry.
2. @8866cf2 BlobGate actor: per-hash PUT serialization (TOCTOU on offset==staged) + client skip-not-abort uploads.
3. @4001452 client URLSession 60s default timed out 5s before server finished → 300s.
4. @e071c4d SIGPIPE from resetting control-plane pollers killed the daemon → SO_NOSIGPIPE + SIG_IGN.
5. @9720665 dead-mount open() wedged upload loop (rdmpw3275m OrbStack NFS, 2.5h) → mount filter at upload time (5th kernel-blocking class closed).
6. @e23245b+@85c1cf2+@656c9c2 (#20 family): respond-early manifest handling + set-based CAS diff + admission control. Measured 236s → 7.2s on identical manifest.
7. @dd78da3 CAS root self-excludes from Spotlight (.metadata_never_index).
8. install_service.zsh: role preserved on bare rerun (two passes — first grep matched a plist comment and flipped all clients to servers at 03:15; PlistBuddy fix; all clients force-reinstalled --client 03:22).

Verification evidence

• Drain: 11,377 → 59,212 blobs; peak ~2,500/min sustained (03:24–05:15).
• 4 of 5 clients fully drained (pending 1–58); rdmpw3275m ~58k remaining.
• Manifest handling: public log line "manifest handled (bytes, Ns)".
• replicantDB shared watchdog caught a real P0 (client names in stale dist build, 22 strings) — verified + fixed by replicantdb-31; scan method corrected fleet-wide (strings|grep with word boundaries, never raw grep on Mach-O).

Undo

• Each fix is one commit in ~/dev/apps/Tyrell (git revert ; redeploy via scripts/deploy_fleet.zsh). Spotlight marker: rm ~/TyrellStore/.metadata_never_index.

Outstanding owner actions (Rich)

• Consider tmutil addexclusion ~/TyrellStore (TM backing up reconstructible CAS).
• contactsd churn recurred (~40–55% CPU) — earlier tied to iPhone simulator; recheck.
• Morning queue in ~/dev/apps/Tyrell/SESSION-STATE.md + QA-CHECKLIST.md.