rdmsm4x-changelog-20260903-2131-remote-control-403-stale-bridge-records

[2026-09-03 21:31:34 EDT · rdmsm4x] · session 158c9525 (notes-republish-pending-changelog) · work window 20:51–21:31 EDT

Summary: /rc ("Remote Control") failed with HTTP 403 in every resumed Claude Code conversation on the fleet because the Remote Control server moved new sessions to an "elevated" security tier this afternoon and refuses to re-attach the older "standard" ones; cleared the stale reconnection records on five Macs with a new script so resumes mint fresh sessions, proved the fix on rdmsm4x, filed INC-20260903-03.

Scope

• Hosts changed: rdmsm4x (hub), rdmbair13m5, rdmbair15m5, rdmpw3265m, rdmpw3275m.
• Not changed: jdmbair13m5 (SSH Permission denied (publickey), needs a console login first; script not installed there yet).

Root cause (verified)

• Symptom line in the transcript: Remote Control disconnected — Remote Control server rejected the request (HTTP 403) — run /remote-control to retry. First seen 20:02:54 EDT; every /remote-control retry repeated it. Fleet Macs all have remoteControlAtStartup: true, so every resumed conversation failed.
• Debug log of a resume (claude --resume <id> --debug-file …): [bridge:repl] Reattaching to persisted bridge session cse_… (fresh-mint fallback, restored_owner_match) → [code-session] /bridge failed 403: Permission denied (source=origin). The CLI then stops; the announced fresh-mint fallback does not fire on a 403.
• API GET /v1/code/sessions/<id> shows every session created before ~13:00 EDT at security_tier: standard and every session created from at least 17:11 EDT at security_tier: elevated. POST …/bridge on an elevated session with only the OAuth bearer answers elevated session requires a trusted device (resource: untrusted_device); on a standard session it answers a bare Permission denied. Claude Code presents a device attestation (DEVICE_ATTESTATION_STATUS_VERIFIED_KEYLESS_DEVICE in the event stream) so new sessions work.
• Positive controls: claude remote-control in ~/dev/fleet and claude --remote-control in ~/dev both registered and connected at once (tier elevated). Account state is fine: claude auth status logged in, max plan, token valid, status.claude.com all operational.
• Not the cause: OAuth scopes (unchanged, includes user:sessions:claude_code), proxy/VPN (none), Claude Code release notes 2.1.253–2.1.260 (no Remote Control auth change).

Fix

• New ~/scripts/claude_rc_reset.zsh v1.0.0 (sha256 17e7fb3cce50fdca…), installed on the five reachable Macs and snapshotted to ~/dev/fleet/tooling/host-scripts/<host>/.
  • check: classifies every conversation that carries a bridge-session record by asking the API for the session's tier (read-only).
  • apply [<sid>…]: for idle conversations whose record is standard, copies the transcript to ~/.claude/backups/rc-reset-<stamp>/<project>/ and appends Claude Code's own clear record {"type":"bridge-session","sessionId":…,"bridgeSessionId":"","lastSequenceNum":0} (the same line the CLI's clearBridgeSession() writes), then re-reads the last record to verify. Running conversations are refused because the CLI rewrites the record every turn. Unknown API answers fail closed.
  • after-exit <pid> <sid>: waits for a running conversation to exit, then clears it. CLAUDE_RC_RESET_ASSUME_STALE=1 lets a watcher armed over SSH (no keychain) proceed on the caller's classification.
  • Over SSH the hub token is piped through stdin (T=$(cat); CLAUDE_RC_RESET_TOKEN="$T" …); the value is never printed.
• Proof on rdmsm4x: after clearing conversation 27c29282, claude --resume logged [remote-bridge] Created session cse_01DwcRJXGrHkhMne43Rm8EE8, v2 transport connected, Flushing 200 history events; the API reports that session elevated.

Numbers

Commands run (representative)

~/scripts/claude_rc_reset.zsh check
~/scripts/claude_rc_reset.zsh apply 27c29282        # test conversation
~/scripts/claude_rc_reset.zsh apply                 # all idle stale on the host
nohup ~/scripts/claude_rc_reset.zsh after-exit 3241 158c9525-… > ~/Library/Logs/claude-rc-reset/after-exit-3241.log &
printf '%s' "$TOK" | ssh <spoke> 'T=$(cat); CLAUDE_RC_RESET_TOKEN="$T" /Users/richh/scripts/claude_rc_reset.zsh apply'

Verification

• check after apply: rdmsm4x stale=3 (all three running), rdmbair13m5 stale=1 (running), rdmbair15m5 stale=2 (running), rdmpw3265m stale=0, rdmpw3275m stale=0.
• Script sha256 matches on all five hosts (17e7fb3cce50fdca).
• Watcher processes confirmed alive with the intended arguments on all three hosts; logs show waiting for pid ….
• Spoke resume mint could NOT be exercised over SSH (Claude Code has no usable login without the console keychain: [bridge:repl] Skipping: bridge not enabled). The record clearing is verified there; the mint is proven on rdmsm4x only.

Outstanding (owner: Rich)

• Exit and resume the running conversations to pick up the fix: on rdmsm4x the sessions named notes-republish-pending-changelog, dev-49, dev-33; on rdmbair13m5 pid 20323; on rdmbair15m5 pids 72112 and 31740. The watcher clears the record within seconds of the exit; the next claude --resume starts Remote Control fresh with history.
• jdmbair13m5: log in at the console once, then from rdmsm4x: ssh jdmbair13m5 'cat > /Users/richh/scripts/claude_rc_reset.zsh' < ~/scripts/claude_rc_reset.zsh and run check / apply with the piped token as above.
• The always-on net.dataroo.claude-rc service on rdmsm4x still serves standard-tier sessions and is connected (phone presence live at 21:00). Its children refreshed tokens after the tier change without dropping, so it was left alone. If it does drop: bash ~/scripts/claude_rc_service.sh restart.
• Ticket INC-20260903-03 stays open until check reports stale=0 on all six hosts.

Traps met this session (recorded)

• scp -q file host:scripts/x returned 0 and wrote nothing on four spokes; ssh host 'cat > /abs/path' < file plus a remote shasum is the install method that verifies itself.
• zsh does not word-split set -- $pair; the first watcher launch started with one argument (memory zsh-no-word-splitting).
• The interactive shell here has grep aliased to a missing ugrep and NO_BARE_GLOB_QUAL set; use /usr/bin/grep and explicit paths.

Backups / undo

• Every modified transcript has a same-name copy under ~/.claude/backups/rc-reset-<stamp>/<project>/ on the host that was changed. Undo = copy it back (or delete the single appended last line).
• No secrets in this record. The OAuth token was read from the login keychain into a shell variable and sent only as a request header.