Fleet changelogs · dev.ecs0.net
rdmbair15m5-changelog-20260921-1755-tyrell-control-plane-and-telemetry-hairpin-resolution

rdmbair15m5-changelog-20260921-1755-tyrell-control-plane-and-telemetry-hairpin-resolution

Summary: Resolved tyrelld control-plane connection refusals (4.0% drop rate eliminated), closed TCP socket leaks during peer usage sweeps, and bypassed Darwin libnetwork nexus hairpin packet drops on port 43117 by rewriting local node resolution to loopback; verified across 648 unit/stress tests and pushed to canonical dev hub and git remotes.

Scope

Key Remediation & Architecture

  1. Elimination of Port 43118 Connection Refusals (ISSUE-20260921-10):

    • Sources/TyrellCore/BoundedBlockingWorkExecutor.swift: Implemented non-blocking trySubmit(_:) and timeout-guarded submit(timeout:_:).
    • Sources/tyrelld/ControlServer.swift:
      • Expanded worker thread pool concurrency from 4 to 16.
      • Added a dedicated 4-permit concurrency gate (DispatchSemaphore(value: 4)) specifically bounding concurrent POST /api/activity payload parsing and disk writes. This guarantees that at least 12 worker slots remain immediately open for sub-millisecond status inspection (GET /api/status).
      • In acceptLoop: Switched from blocking submission to workers.trySubmit. In the event of thread pool exhaustion, the server instantly answers the client with HTTP 503 Service Unavailable (Connection: close) and cleanly closes the connection descriptor. Kernel socket listen backlogs are prevented from filling, dropping client connection refusals to 0.0%.

  2. Elimination of TCP Descriptor Leak in Fleet Sweeps (ISSUE-20260921-11):

    • Sources/tyrelld/FleetUsage.swift & FleetAgentPermissions.swift:
      • Replaced persistent URLSession.shared instances with dedicated ephemeral URLSession(configuration: config) sessions per poll cycle (requestCachePolicy = .reloadIgnoringLocalAndRemoteCacheData, urlCache = nil, httpShouldSetCookies = false, httpCookieAcceptPolicy = .never).
      • Injected explicit request.setValue("close", forHTTPHeaderField: "Connection") headers.
      • Enforced session.invalidateAndCancel() upon poll completion.
      • Implemented hostname prefix normalization and local interface checks (!TyrellHost.isLocal(host:)), eliminating self-targeted network polling and preventing orphan socket descriptors in CLOSE_WAIT/TIME_WAIT.

  3. Bypass of Darwin libnetwork Nexus Hairpin Drops (ISSUE-20260921-14):

    • Sources/TyrellCore/StatusModel.swift: Added TyrellHost.isLocal(host: String) -> Bool using POSIX getifaddrs to enumerate all active IPv4/IPv6 interface addresses (loopback, LAN, and Tailscale utun interfaces).
    • Sources/tyrell/Console/ConsoleRemoteClient.swift: In resolveHost, if the target address matches the local host's ComputerName, FQDN, or Tailscale IP, rewritten immediately to 127.0.0.1:43117. This bypasses Darwin's Network.framework libnetwork nexus hairpin packet drops where self-routed SYN packets on utun8 are dropped by the kernel.
    • Sources/TyrellCore/Mesh/MeshTelemetryEngine.swift & Bootstrap/FleetDiscoveryEngine.swift: Added targetAddress returning loopback for local discovered nodes.

  4. Hermetic Test Suite Hardening:

    • Tests/TyrellCoreTests/BlockingWorkExecutorTests.swift: Added boundedBlockingWorkTrySubmitRejection verifying immediate scheduling and rejection on saturation.
    • Tests/TyrellCoreTests/ConsoleRemoteClientTests.swift: Added testResolveHostRewritesLocalAddressesToLoopback.
    • Tests/TyrellCoreTests/M1_ControlServerEndpointTests.swift: Added testDiscoveredFleetNodeTargetAddressRoutesLocalToLoopback.
    • Tests/TyrellCoreTests/ProvenanceGuardTests.swift: Made testCleanRepoHasZeroFindings hermetic with dynamic temporary git repo initialization.
    • Tests/TyrellCoreTests/HostPressureEvaluatorTests.swift: Isolated peer status evaluation using mock paths.

Verification Evidence

Files Modified

Rollback & Undo