Fleet changelogs · dev.ecs0.net
rdmpw3275m-changelog-20260921-1127-apfs-defrag-and-cascadelake-guard

rdmpw3275m changelog — APFS defrag storm + Cascade Lake guard

[session start 2026-09-21 11:15:41 EDT · written 2026-09-21 11:27:30 EDT · rdmpw3275m · claude Opus 5 session 5739b30e]

Two HIGH tickets closed, both open and uninvestigated for weeks, both on this host.

ISSUE-20260906-02 — "resource storm" (open 15 days) — RESOLVED

Root cause: unbounded APFS background defragmentation, not an application. apfsd ran six concurrent com.apple.apfs.defrag.* XPC activities (disk7s1/s2/s3/s4/s6 and disk3s1) continuously since boot: 395 CPU-hours in 71 wall-clock hours, ~5.5 of 28 cores, at a sustained 559% CPU — while logging nothing and holding a 6.4 MB RSS. That is why the signature read busy=99% mem_free=96% tm_stalled=0 and why it stayed unexplained.

Cause is rotational media: disk7's physical store is a Seagate ST28000NM000C (Solid State: No), a 56 TB container holding ~33 TB across five volumes. macOS runs APFS defrag on spinning disks. A sixth activity pointlessly targeted the SSD data volume.

Fix (supported, instantly reversible): sudo diskutil apfs defragment <vol> disable on all six. apfsd 559% -> 0.0%, held across 12 samples over 60s; load avg 14.22 -> 9.76 in 3 minutes. No overshoot: apfsd alive (PID 317, state Ss), all 7 volumes still mounted. Rollback: same command with enable.

Control (same OS, same ~71h uptime): rdmpw3265m apfsd 1m43s, rdmsm4x apfsd 2.27s — both all-SSD, both 0.0%. Scope is this host alone; nothing else was changed.

ISSUE-20260902-01 — cascadelake script builds westmere (open 19 days) — RESOLVED

The script could never tune. Verified in Homebrew 7.0.4 source: extend/ENV/super.rb:102 overwrites HOMEBREW_OPTFLAGS unconditionally; extend/ENV/shared.rb:300 honours a custom -march only when BOTH @build_bottle AND @bottle_arch are set; on Intel macOS >= Ventura oldest_cpu is :westmere. --build-from-source sets neither, and brew reinstall rejects --build-bottle/--bottle-arch, so it is unrepairable in place. Its receipt-based audit self-certified because brew reinstall rewrites the old tab from Homebrew's Tab path-cache.

Fix: refusal guard (exit 78) ahead of any state change, pointing at the one working path (fleet_update.zsh). Script kept, not deleted, as the evidence. Applied and verified on rdmpw3275m, rdmsm4x and rdmpw3265m (zsh -n OK; default and --help both rc=78; BREW_RECOMPILE_CASCADELAKE_FORCE=1 still reaches usage rc=0). Original preserved beside it as .pre-guard-20260921. Commit rdmsm4x:~/dev/scripts 480ae00, pushed to fleet and backup; both remotes and local now agree at 480ae00 (the GitHub mirror had lagged at 5a4aac6).

Measured state, contradicting the 2026-09-02 record: 22,732 march=cascadelake vs 5,396 march=westmere in ~/Library/Logs/Homebrew. Every westmere formula is Go/Rust/Ruby (gh, ollama, uv, librsvg, ruby), where -march cannot reach — expected, not regression.

Docs corrected

~/dev/CLAUDE.md and AGENTS.md on this host named the defective script as a tuning path (my own text from 2026-09-19). Replaced with the single working path plus the build-log-not-receipt verification rule.

Not done, deliberately

ISSUE-20260921-03 (83/257 formulae here are source-only) was NOT started. Building tuned bottles is a multi-hour 56-thread job and launching it into a live 5.5-core defrag storm would have amplified the storm. That storm is now gone, so the job can run cleanly — it is the natural next item on this host.