rdmsm4x — dev.dataroo.net 503: the whole internet shared one rate-limit bucket
2026-08-23, 11:18–11:21 EDT · rdmsm4x
Rich reported a 503. The site was never down. nginx was
rejecting requests from its own rate limiter, and the limiter was
counting every visitor on Earth as one client.
Root cause
~/dataroo.net/nginx_auth.conf trusted
172.16.0.0/12 for set_real_ip_from. Compose
had put this stack on 192.168.107.0/24, with
cloudflared at 192.168.107.3 — outside that range.
So real_ip_header CF-Connecting-IP was never
applied. $binary_remote_addr stayed the tunnel's
own address, which limit_req_zone keys on. One 15
r/s bucket for the entire internet — every visitor, tab, and
polling client drawing from the same allowance.
It survived every health check either agent ran because a sequential probe cannot exhaust it. Only concurrency does.
| Before | After | |
|---|---|---|
| 60 concurrent | 32× 401, 8–18× 503 | 32× 401, 28× 429 (correctly labelled, single source) |
| Access log client | 192.168.107.3 on every line |
69.124.66.186 — a real visitor IP |
| 8 concurrent tabs, authenticated | would 503 | 8× 200 |
| Page + poll ×6, authenticated | — | 12× 200 |
| Second host concurrently | competed for the same bucket | 401 auth challenge, not 429 |
That last row is the proof the limit is now per visitor: a second source is unaffected by the first exhausting its allowance.
Changes
set_real_ip_from 192.168.107.0/24;added — the network cloudflared is actually on.set_real_ip_from 172.16.0.0/12;kept, so a recreated Compose network landing in Docker's default pool does not silently reintroduce the fault.limit_req_status 429;added. The default 503 sent two people hunting a dead container for a limiter working exactly as configured. A rejection must be legible as what it is.
A second defect, found while applying the first
nginx -t failed with
open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)
immediately after the edit — on a container that was up and serving.
Docker bind-mounts a single file by inode. An editor
that writes a temp file and renames it — which is most of them — gives
the file a new inode, and the container's mount still points at the old,
now-unlinked one. nginx kept serving from its in-memory config, so the
site stayed up and only reload broke.
docker compose restart auth_proxy re-resolves the
mount.
Anything bind-mounting an individual file has this property. Mount the directory, or restart after editing.
Verification
Config validated inside the container before reload,
the new directives confirmed present in the container's own copy, then
tested against the live tunnel — not against 127.0.0.1,
which never traverses the path that failed.
Undo
~/dataroo.net/nginx_auth.conf.bak-20260823-*, then
docker compose restart auth_proxy.
Credit
Reproduced and root-caused by claude@rdmbair15m5, which correctly declined to edit a live service config on another host's behalf. Verified independently here before any change: the CIDR containment, the logged client IP, and the concurrency-only failure were each measured, not accepted.
Recommended, not done
Pin the Compose network subnet in docker-compose.yml so
it cannot drift at all. It requires recreating the network, which
briefly drops the site — Rich's call, not an overnight
action.