rdmsm4x-changelog-20260827-0654-litellm-qa-agent

Implemented and accepted a disposable, credential-free OrbStack QA agent for the approved loopback LiteLLM route, and restored the accepted LiteLLM service on 127.0.0.1:4000 so local development no longer encounters a closed health endpoint.

Scope

• Host changed: rdmsm4x only.
• Canonical project: /Users/richh/dev/localAI/litellm.
• Isolated implementation worktree: /Users/richh/dev/_worktrees/codex-litellm-qa-agent-20260827.
• Branch: codex/litellm-qa-agent-20260827.
• Implementation commit: 7ee41850cf5c29ec53fc5200170de4f582856360.
• Milestone-state commit: aceaf5389ef82255c7749bb3fa28555feb6f6af3.
• Runtime left active: accepted litellm-loopback-canary only, healthy on 127.0.0.1:4000.
• Runtime explicitly untouched: PostgreSQL, Mem0, Qdrant, OpenRouter/provider accounts, Tyrell, Buzz, remote fleet runtimes, and real client profiles.

Files changed

• /Users/richh/dev/localAI/litellm/compose.qa-agent.yaml
• /Users/richh/dev/localAI/litellm/qa/litellm_qa_agent.py
• /Users/richh/dev/localAI/litellm/scripts/litellm-qa.sh
• /Users/richh/dev/localAI/litellm/tests/test_qa_agent.py
• /Users/richh/dev/localAI/litellm/tests/validate-qa-static.zsh
• /Users/richh/dev/localAI/litellm/README.md
• /Users/richh/dev/localAI/litellm/SESSION-STATE.md
• /Users/richh/dev/localAI/litellm/ISSUES.md
• /Users/richh/dev/localAI/litellm/docs/architecture-and-routing-design-2026-08-26.md
• /Users/richh/dev/localAI/litellm/docs/qa-agent-evidence-2026-08-27.md
• /Users/richh/dev/localAI/litellm/docs/superpowers/specs/2026-08-27-orbstack-qa-agent-design.md
• /Users/richh/dev/localAI/litellm/docs/superpowers/plans/2026-08-27-orbstack-qa-agent.md
• /Users/richh/.agent-coordination/checkins/codex-litellm-qa-agent-20260827-0631.md
• /Users/richh/.agent-coordination/checkins/codex-litellm-projects-index-20260827-0658.md
• /Users/richh/Library/Mobile Documents/com~apple~CloudDocs/Codex/projects/AGENTS.md principles/PROJECTS.md
• This changelog.

All project files are committed only on the isolated branch at the time of this record. Generated metadata reports remain under the ignored reports/ directory.

What changed

• Started the already accepted, pinned LiteLLM canary with its existing operator and intentionally left it healthy on IPv4 loopback.
• Added one non-listening QA service using the same immutable LiteLLM image, a read-only root filesystem, dropped capabilities, no-new-privileges, fixed local route arguments, and only code/report bind mounts.
• Added a Python-standard-library CLI agent that checks liveliness, exact one-alias model discovery, non-streaming chat envelopes, and streaming termination.
• Added a ten-key metadata-only report allowlist, bounded HTTP reads, categorized errors, and safe-character/length filtering for response identifiers.
• Added a self-locating Bash operator for preflight, up, run, qa, status, and down.
• Added exact-project cleanup for the transient QA container and private network on normal completion, failure, or signal.
• Updated project architecture so clients call the separate Mem0 service over MCP/API; LiteLLM never uses Mem0 as a prompt/query cache.
• Recorded the dev.dataroo.net source/runtime/publisher boundary without editing or publishing that project.
• Added exactly one LiteLLM milestone line to the canonical operating project index after fleet sync, collision recheck, a narrow lease, and a checksum-verified backup.

Commands run

• bash scripts/litellm-canary.sh preflight
• bash scripts/litellm-canary.sh up
• bash scripts/litellm-canary.sh verify
• python3 -m unittest -v tests/test_qa_agent.py
• python3 -m py_compile qa/litellm_qa_agent.py tests/test_qa_agent.py
• zsh tests/validate-qa-static.zsh
• zsh tests/validate-static.zsh
• bash -n scripts/litellm-qa.sh
• docker compose --project-name litellm-qa-agent --project-directory "$PWD" -f compose.qa-agent.yaml config --quiet
• bash scripts/litellm-qa.sh qa
• Repeated bash scripts/litellm-qa.sh run live acceptance runs.
• Read-only curl, docker inspect, docker ps, docker network ls, jq, and log-marker checks.
• gitleaks dir --redact --no-banner --no-color --exit-code 1 .
• git diff --check, explicit-path git add, and git commit.

Verification evidence

• Original LiteLLM canary contract: 60 checks passed, 0 failed.
• QA behavioral suite: 6 tests passed, including extra-model rejection and unsafe response-ID redaction.
• QA rendered-Compose/operator contract: 10 checks passed, 0 failed.
• Four consecutive live QA reports: 20 total events, 0 failures; every report used the fixed metadata-only schema.
• Final exact-code live report: reports/qa-20260827T104820Z-85283.jsonl.
• Final transient state: zero QA-agent containers and zero QA-agent networks.
• LiteLLM state: container running/healthy; /health, /health/liveliness, and /v1/models returned HTTP 200; exact host publication 127.0.0.1:4000; exact model surface agent-local only.
• Secret scan: no leaks found; no disclosed provider credential was read, copied, stored, or used.
• Git: committed branch clean at aceaf5389ef82255c7749bb3fa28555feb6f6af3; implementation commit remains 7ee41850cf5c29ec53fc5200170de4f582856360.

Backups and preservation

• No user database, provider account, secret store, model store, client configuration, or production website file was modified, so no new data backup was required for this bounded change.
• The existing LiteLLM PostgreSQL service and volumes were not connected or altered.
• Existing canary rollback evidence and source-reconciliation archives remain preserved.
• Canonical index backup: /Users/richh/dev/_backups/litellm-projects-index-20260827-0658/PROJECTS.md.before, SHA-256 941c28089e5185d307f46012f106bea11f6954efaee8628851db024e2f62875e.
• The isolated worktree and branch remain intact for review/integration.

Undo

To stop only this accepted local LiteLLM/QA lane and close TCP 4000:

bash /Users/richh/dev/_worktrees/codex-litellm-qa-agent-20260827/scripts/litellm-qa.sh down

To leave the implementation unintegrated, keep branch codex/litellm-qa-agent-20260827 and its worktree as-is. No remote push or merge occurred in this task.

Outstanding owner actions

• Choose whether to merge the isolated branch to main, push it for review, or keep it as-is.
• Revoke and replace the provider credential exposed in chat before any cloud/provider request.
• Back up and restore-test the existing PostgreSQL data before database-backed virtual-key or budget work.
• Approve exact per-harness budgets and each real-client canary separately.
• Keep dev.dataroo.net publication blocked until its active production writer explicitly hands off and a separate publish approval is given.
• Apple Notes mirror is pending. The shared Notes publisher was deliberately not invoked because its documented count/deletion hazard remains unresolved.