rdmsm4x-changelog-20260827-0518-litellm-loopback-canary

Implemented and verified the approved LiteLLM credential-free loopback canary so the local routing design has a reversible, content-free acceptance path before any provider, database, key, or client work.

Scope

• Host: rdmsm4x
• Canonical project: /Users/richh/dev/localAI/litellm
• Task worktree: /Users/richh/dev/_worktrees/codex-litellm-loopback-canary-20260827
• Branch: codex/litellm-loopback-canary-20260827
• Mode: Production
• Runtime boundary: disposable local-only LiteLLM proxy to the already-installed local Ollama model; success must end stopped
• Explicit exclusions: credentials, provider accounts, key creation or rotation, database connection or mutation, client routing, public or tailnet exposure, purchases, subscription changes, and external communications

Project files changed

• /Users/richh/dev/localAI/litellm/ISSUES.md
• /Users/richh/dev/localAI/litellm/README.md
• /Users/richh/dev/localAI/litellm/SESSION-STATE.md
• /Users/richh/dev/localAI/litellm/compose.canary.yaml
• /Users/richh/dev/localAI/litellm/config/litellm.local-canary.yaml
• /Users/richh/dev/localAI/litellm/config/litellm.production-policy.example.yaml
• /Users/richh/dev/localAI/litellm/docs/architecture-and-routing-design-2026-08-26.md
• /Users/richh/dev/localAI/litellm/docs/canary-evidence-2026-08-27.md
• /Users/richh/dev/localAI/litellm/docs/superpowers/plans/2026-08-27-loopback-canary.md
• /Users/richh/dev/localAI/litellm/scripts/litellm-canary.sh
• /Users/richh/dev/localAI/litellm/tests/validate-operator.sh
• /Users/richh/dev/localAI/litellm/tests/validate-static.zsh

The project was initialized as a standalone Git repository. Baseline main commit 6bd49b71103d8a0b03365e919050372235f20319 preserved the approved documentation checkpoint; implementation was isolated in the linked task worktree and branch above.

The exact implementation and evidence tree was committed as e9a837b03cf7d9a894f51b493be83b9f9c44d041. A final independent read-only review returned CLEAN, reproduced the static/syntax/whitespace checks, confirmed the platform-aware ARM64 index/child image policy, and found no remaining Critical or Important blocker in scope.

The accepted branch was then fast-forwarded, without a merge commit or history rewrite, into canonical main at 74caa92e3bacae28b33d61cd16140204099ad70a. Canonical-root verification showed a clean working tree, 60 passing static/structural checks, valid syntax and Compose rendering, no redacted-scan leak, no canary container/network/listener, and the pre-existing PostgreSQL container still healthy.

Runtime and configuration changes

• Added one Compose service named litellm-loopback-canary using LiteLLM v1.98.0 at an immutable multi-architecture image digest.
• Published only 127.0.0.1:4000:4000 during the bounded canary.
• Exposed exactly one alias, agent-local, mapped only to the installed ollama_chat/glm-4.7-flash:q4_K_M model through the Docker host gateway.
• Used a read-only configuration mount and root filesystem, no-new-privileges, all capabilities dropped, a temporary filesystem, no persistent volume, and no external DNS resolver in the canary container.
• Disabled raw request/response and message logging, disabled prompt storage in spend logs, and enabled exception/API-key-info redaction.
• Added an operator that is self-locating, refuses root execution, validates platform-aware image identity, records only metadata, and automatically rolls back failures and signals.
• Added static parsed-structure tests plus injected failure/signal rollback tests.
• Cached the verified LiteLLM Linux/ARM64 image locally. This is the only persistent runtime artifact.

The existing local-llm-postgres container was neither connected to nor changed and remained healthy. No provider key, master key, database URL, cloud model, fallback, client profile, secret store, or provider account was accessed.

Commands run

• bash -n scripts/litellm-canary.sh
• bash -n tests/validate-operator.sh
• zsh -n tests/validate-static.zsh
• zsh tests/validate-static.zsh
• bash scripts/litellm-canary.sh __test-image-digests
• docker compose --project-name litellm-local-canary -f compose.canary.yaml config -q
• gitleaks dir . --redact --no-banner --exit-code 1
• git diff --check
• bash tests/validate-operator.sh
• bash scripts/litellm-canary.sh canary
• bash scripts/litellm-canary.sh status
• Bounded docker ps, docker network ls, docker image inspect, docker inspect, lsof, curl, and jq checks used by the operator or final verification

Verification evidence

• Static validator: 60 checks passed, 0 failed.
• Image policy self-test: accepted only Linux/ARM64 with the approved multi-architecture index or approved ARM64 child manifest; rejected a wrong digest and wrong architecture.
• Secret scan: no leaks found in the project tree; output was redacted.
• Operator integration run: /Users/richh/dev/_worktrees/codex-litellm-loopback-canary-20260827/reports/operator-test-20260827T091619Z-63057/.
• Injected standalone verify failure: returned 1 and rolled back.
• Injected standalone restart-verify failure: returned 1 and rolled back.
• TERM during the armed verification hold: returned 143 and rolled back.
• Final live report: /Users/richh/dev/_worktrees/codex-litellm-loopback-canary-20260827/reports/canary-20260827T091738Z-64404.jsonl.
• Final live run: 18 metadata-only stages passed, 0 failed, from 2026-08-27T09:17:38Z through 2026-08-27T09:18:09Z.
• Model listing exposed exactly agent-local.
• Initial local chat returned HTTP 200 in 2,089 ms; repeat chat after restart returned HTTP 200 in 153 ms; response bodies were validated transiently and discarded.
• Streaming completed with 18 data events plus its terminal event; the stream body was discarded.
• Unique non-secret markers were absent from ordinary container logs and persisted reports.
• The unrelated running-container name/image/state set was unchanged across the complete canary.
• Final state: canary container absent, canary network absent, TCP 4000 closed, and pre-existing PostgreSQL healthy.

Before, after, and rollback

• Before: approved documentation checkpoint and no executable proxy canary.
• After: standalone project repository with a pinned, secret-free local canary, policy example, operator, tests, compact session state, and durable metadata-only evidence. The proxy is stopped.
• Persistent delta: project Git/files and the cached LiteLLM Linux/ARM64 image only.
• Immediate runtime rollback command: bash /Users/richh/dev/localAI/litellm/scripts/litellm-canary.sh down.
• Source rollback: revert the relevant Git commits in a new recovery branch; do not reset or overwrite the canonical tree.
• The cached image may be removed later only through a separately reviewed Docker cleanup; it was intentionally retained for repeatable canaries.

Outstanding owner-gated actions

1. Revoke and replace the provider credential previously exposed in chat. The disclosed value was not used, inspected, copied, or saved here.
2. Create and restore-test a recoverable backup of the existing PostgreSQL state before any database connection, schema, virtual-key, or budget work.
3. Select exact per-harness caps within the approved aggregate policy and provide approved secret references before a provider-backed canary.
4. Prove database-backed budget rejection and a maximum 90-day key lifetime with one isolated cloud alias before any real client reroute.
5. Canary each compatible client separately; consumer/subscription lanes remain separate.
6. Mirror this changelog to the rdmsm4x Apple Notes folder only after the shared Notes publisher's unresolved deletion/count hazard is fixed and verified. Notes was not called in this run.