rdmsm4x-changelog-20260904-2121-tyrell-observatory-heatmap-fleet-relay

Fixed Tyrell Observatory heatmap truncation by expanding port 43118 SnapshotBox feeder buffer to 5,000 events and enabled remote spoke daemons to relay fleet activity, deploying the fix across all 6 fleet machines.

1. Scope

• Target Hosts: rdmsm4x (hub), rdmbair15m5 (canary), rdmbair13m5, jdmbair13m5, rdmpw3265m, rdmpw3275m.
• Target Applications/Services: Tyrell.app, tyrelld (daemon), tyrellbar, tyrell CLI, tyrell-mcp.
• Tickets Addressed: ISSUE-20260904-16 (resolved), ISSUE-20260903-31 (resolved).

2. Root Cause & Solution

1. Feeder Truncation (Port 43118):
   • Root Cause: In Sources/tyrelld/Daemon.swift:111, the background feeder periodically updating SnapshotBox called try env.store.recentEvents(limit: 100). Because AppStatusFeed and tyrellbar prioritize port 43118, the UI received only the last 100 events (~2.3 hours), leaving columns 0–31 (the left ~75% of the 24h timeline) permanently blank on the 24h timeline.
   • Solution: Switched the feeder to query up to 5,000 events directly from the in-memory ring buffer: await env.store.activityBuffer.events(limit: 5000).
2. Spoke Node Isolation:
   • Root Cause: Remote spoke daemons (!env.isServer) only write local host events into their SQLite databases, and ClientLoop.swift previously only synced fleet status without relaying fleet activity from the hub. Remote Tyrell.app instances therefore had zero activity data for any host except themselves.
   • Solution: Added ActivityEventBufferActor.mergeFleetEvents(_:) to deduplicate and store relayed events in memory without queueing them for local SQLite insertion. Wired ClientLoop.fetchFleetActivity(base: base) into the 30-second syncLoop. Added master fallback URLs in AppStatusFeed.swift.
3. Codesign Prompt Mitigation (Issue #51):
   • Root Cause: scripts/install_service.zsh unconditionally attempted to sign binaries with local certificates even if already validly signed with Hardened Runtime, provoking unwanted modal SecurityAgent prompts.
   • Solution: Added pre-check codesign -v "$BINARY" to skip signing on already-signed universal artifacts.

3. Exact Files Touched

• Sources/tyrelld/Daemon.swift: Replaced limit: 100 with await env.store.activityBuffer.events(limit: 5000).
• Sources/TyrellCore/Activity/ActivityEventBufferActor.swift: Added mergeFleetEvents(_:) for in-memory multi-host event aggregation.
• Sources/tyrelld/ClientLoop.swift: Added fetchFleetActivity(base:) and wired master event relaying into syncLoop.
• Sources/tyrell-app/AppStatusFeed.swift: Added dynamicActivityURLs with master fallback and 5,000-event retention cap.
• Tests/TyrellCoreTests/ActivityEventBufferActorTests.swift: Added unit test mergeFleetEventsCombinesHostsWithoutQueueingUnpersisted.
• scripts/install_service.zsh: Added codesign -v gate to prevent redundant signing prompts.
• ISSUES.md: Documented and closed Issue #51 and Issue #52.

4. Commands Run

• swift test --filter ActivityEventBufferActorTests: Verified 6/6 tests passed.
• swift build -c release: Rebuilt production binaries.
• zsh scripts/make_app_bundle.zsh: Assembled universal2 (arm64 + x86_64) signed Tyrell.app bundle and hardened runtime companion tools.
• scripts/install_service.zsh --server: Re-installed and restarted local hub tyrelld.
• sudo -n /bin/zsh scripts/install_app_local.zsh --bundle .build/Tyrell.app --target-user richh: Installed updated Tyrell.app locally on rdmsm4x.
• scripts/install_tyrellbar_agent.zsh: Verified local menu bar agent running from bundle.
• zsh scripts/redeploy_app_fleet.zsh --execute-canary: Staged and installed on canary rdmbair15m5.
• zsh scripts/run_tyrell_app_canary_probe.zsh ...: Executed canary verification probe (passed).
• ssh [email protected] ... issue_tyrell_app_canary_receipt.zsh ...: Issued cryptographic canary receipt.
• zsh scripts/redeploy_app_fleet.zsh --execute-fleet --deployment-nonce ...: Staged and deployed Tyrell.app to all fleet hosts (rdmpw3275m, rdmbair13m5, jdmbair13m5, rdmpw3265m).
• scripts/deploy_fleet.zsh --hosts rdmbair15m5: Deployed daemon, CLI, and MCP to canary.
• scripts/deploy_fleet.zsh --hosts rdmbair13m5,jdmbair13m5,rdmpw3265m,rdmpw3275m: Deployed daemon, CLI, and MCP to remaining 4 spoke nodes.
• ticket resolve ISSUE-20260904-16 ...: Closed primary issue.
• ticket resolve ISSUE-20260903-31 ...: Closed related issue.

5. Verification Evidence

• Test Suite: ActivityEventBufferActorTests completed with 6 passed tests in 0.015s.
• Canary Probe: probe_state=passed, receipt_state=accepted on rdmbair15m5.
• Live Fleet Port 43118 Query (Post-Deployment):
  • rdmsm4x: 2,580 events across all 6 hosts (jdmbair13m5, rdmbair13m5, rdmbair15m5, rdmpw3265m, rdmpw3275m, rdmsm4x).
  • rdmbair15m5: 2,949 events across all 6 hosts.
  • rdmbair13m5: 2,917 events across all 6 hosts.
  • jdmbair13m5: 2,816 events across all 6 hosts.
  • rdmpw3265m: 2,777 events across all 6 hosts.
  • rdmpw3275m: 2,937 events across all 6 hosts.
• Timeline Coverage: Events span 259+ hours of active history, ensuring 100% population of the 36-column Observatory heatmap with no blank margins.

6. Backup & Rollback

• Automatic rollbacks preserved under /Applications/.tyrell-rollbacks/<timestamp>-<hash>/Tyrell.app on all hosts.
• To rollback app on any host: replace /Applications/Tyrell.app with the archived rollback bundle and execute sudo killall Tyrell && open /Applications/Tyrell.app.
• Git commit hash on canonical main: f88ae25. Undo via git revert f88ae25.

7. Outstanding Owner Actions

• None. All services and GUI apps are running, fully healthy, and verified live across the fleet.