Zero Trust resilience & fallback¶

What to do when Cloudflare Zero Trust stops working.

All servers run zt_lockdown_enabled: true. The tunnel is the only normal way in — and on web hosts, it is also the path customer traffic takes. See Cloudflare Zero Trust — lockdown mode for how lockdown is configured and applied.

Scope¶

This is a technical runbook. If the outage turns out to be a security incident rather than a failure, run it alongside Incident Response. For an outage that lasts long enough to need business workarounds, see the IT Contingency Plan.

How lockdown is enforced¶

Under lockdown, sshd listens only on 127.0.0.1, UFW denies inbound traffic, and cloudflared publishes SSH (localhost:22) and HTTP (localhost:80) to Cloudflare.

Enforcement happens in two layers with different reach:

Docker writes directly to the DOCKER-USER iptables chain and bypasses UFW, so published container ports are closed by the Hetzner firewall alone. Changes to the Hetzner firewall policy are therefore as security-critical as changes to the firewall_ufw role.

Architectures¶

Failure scenarios¶

Two things worth knowing before an incident:

Customer traffic runs on BYPASS-GROUP-PUBLIC-WEB and does not depend on our identity provider. Scenario 3 costs admin access and CI deploys, not revenue. Changing DNS in that scenario adds risk and fixes nothing.

Scenario 3 hits every host at once. Procedure A is per host, so recovering the full estate takes roughly 10 minutes per server. Prioritise the hosts you actually need rather than working through all of them.

Database hosts have no customer-facing path. Only Procedure A applies, and it is sufficient — restores run over SSH once access is back.

When to act¶

One clock. It starts when customer impact is confirmed, not when the first alert fires, and it does not reset between procedures.

45 minutes is a single number rather than a range, so nobody debates the threshold mid-incident. The incident lead may move it explicitly — earlier during peak sales hours, later overnight. The same number applies to every brand; we accept mild suboptimisation to keep the decision simple.

All three conditions must hold before starting Procedure B:

1. Impact is confirmed by an actual request against the affected endpoint
2. Root cause is the tunnel or cloudflared, not the Cloudflare edge
3. Cloudflare status shows no global incident. If there is one, the bypass will not help and the DNS API may itself be unavailable

Incident lead: the first developer to confirm impact holds the role until they explicitly hand it over. The lead decides whether Procedure B runs. Without a named owner, the usual failure mode is that either nothing happens or everything happens at once.

Procedure A — restore access to a server¶

Hetzner Console is out-of-band, at a different provider, with credentials stored separately in Dashlane. That independence is what makes it usable even when Cloudflare or our identity provider is the problem.

1. Open Hetzner Console for the affected host and log in as root.

2. Diagnose:

systemctl status cloudflared
journalctl -u cloudflared -n 100 --no-pager

3. Restart before anything else. This resolves most cases without reopening SSH:

systemctl restart cloudflared
sleep 15 && systemctl status cloudflared

If the tunnel comes back: log the incident and stop here.

4. Reopen SSH only if real debugging is needed. Restrict it to the office IP:

ufw allow from 87.116.13.2 to any port 22 proto tcp
sed -i 's/^ListenAddress 127.0.0.1/ListenAddress 0.0.0.0/' /etc/ssh/sshd_config
grep '^ListenAddress' /etc/ssh/sshd_config   # must print 0.0.0.0
sshd -t && systemctl restart ssh

The grep matters: if the directive is commented out or missing, sed changes nothing and fails silently. If it prints nothing, add ListenAddress 0.0.0.0 manually before restarting.

If the office VPN is also unavailable, use the NordVPN fallback — see VPN — Backup VPN. Its static IP is a different one (185.183.55.9), so allow that address instead of the office IP in the ufw allow above.

5. Verify from outside before leaving the console. <HOST_IP> is the server's public IPv4 address, shown in the Hetzner panel:

ssh -o ProxyCommand=none <user>@<HOST_IP>

Procedure B — route around the tunnel¶

We keep Cloudflare in front of the site and remove only the tunnel from the path.

Why not point DNS straight at the origin: cloudflared speaks plain HTTP to localhost:80, so the origin holds no valid certificate for the public hostname. Going unproxied would produce certificate errors for customers and drop WAF at the same time. A proxied A record keeps both TLS termination and WAF in place.

1. Identify the record. Look up the brand in the per-host reference for the hostname, zone, public IP and current tunnel UUID. If the table is missing or stale, open the zone in the Cloudflare dashboard and record the existing CNAME target before changing anything — you need it to roll back.

2. Open port 80 in the Hetzner Cloud Firewall. This is the layer that actually closes the port. nginx already publishes on 0.0.0.0:80, and Docker bypasses UFW, so no change on the server itself is required.

Add one inbound rule to the host's firewall policy: TCP port 80, sources set to Cloudflare's IPv4 ranges from https://www.cloudflare.com/ips-v4 (roughly 15 ranges; they fit in a single rule). Do not open to 0.0.0.0/0.

This step's control plane is Hetzner, independent of Cloudflare. We can open the origin even if the Cloudflare dashboard is unavailable — only step 3 requires Cloudflare access.

3. Change the DNS record:

4. Verify. The site responds, and the response carries a cf-ray header. The header confirms WAF is still in the path; without it, the record went unproxied and the site is exposed.

Cloudflare preserves the Host header, so vhost routing inside the container resolves correctly with no further changes.

Known limitation¶

In scenarios 5 and 6, Cloudflare itself is unavailable — and our DNS is hosted there, so we cannot repoint anything. Procedure B does not apply.

True direct-to-origin routing would require certificates pre-installed per hostname on each origin, plus accepting that WAF drops out of the path. That is not in place, and not having it is a deliberate accepted risk. Full coverage would need a secondary DNS provider on the apex domains, which is a separate project.

In these scenarios, the response is communication rather than remediation: notify stakeholders, point to Cloudflare's status page, and wait. Follow the communication cadence in the IT Contingency Plan.

Rollback¶

Skipping this is how servers end up sitting half-open for weeks after an otherwise well-handled incident. Run it the same day.

1. Restore the DNS record to its original CNAME target, if Procedure B was used.

2. Remove the port 80 rule from the Hetzner Cloud Firewall policy.

3. Re-apply lockdown. This rebinds sshd to 127.0.0.1 and restores UFW:

ansible-playbook -i inventories/prod/hosts.yml lockdown.yml --limit <host>

4. Confirm the host has converged. A second run should change nothing — any changed count above zero means manual edits are still in place:

ansible-playbook -i inventories/prod/hosts.yml web.yml --limit <host>
# Expected: changed=0

5. Check for leftover firewall rules from Procedure A:

ufw status numbered

6. Verify WAF rules are intact in the Cloudflare panel — see WAF.

7. Write up what failed, which procedures were used, and the time from detection to restored service. That number is what keeps the 45-minute threshold honest.

Detection¶

Uptime checks run at Cloudflare with response code and response body tests. Two adjustments are needed:

• The body test on the headless site must match something dynamically rendered. Otherwise it passes from cache while the GraphQL backend is down, and we find out late.
• Cloudflare-hosted monitoring shares a failure domain with what it monitors. It covers scenarios 1–4 and is blind to 5 and 6. One external check on a single endpoint closes that gap cheaply.

Alongside this: a Netdata alert on cloudflared service state and tunnel connection count, and a pinned cloudflared version so unattended upgrades cannot break the tunnel without warning.

Alerts must reach a channel that does not depend on Cloudflare — otherwise we lose the alert in exactly the scenarios it exists for.

Testing¶

Record the actual time from detection to restored access. Without that number, the 45-minute threshold is guesswork.

Last tested¶

Overwrite these rows each time — this records the current state, not a history. Full notes belong in the recurring Asana task.

If a date here is more than eight months old, treat the procedure as unverified and expect it to take longer than documented.

Appendix — per-host reference¶

The values Procedure B needs, kept here so nobody has to look them up in a dashboard that may be slow or partially unavailable during an incident.

The tunnel UUID is the rollback value. It is what the DNS record points at today, and what it must point at again afterwards. If it is wrong here, rollback is guesswork.

Database hosts are out of scope for Procedure B and are deliberately omitted.

Where to find each value

Open items¶

• cloudflare_policy_allow_bitbucket_ci refers to a CI system we no longer use. Confirm that GitHub deploys run outside Access, or create an equivalent policy — then rename the variable.
• Redis and cadvisor publish on 0.0.0.0 and rely solely on the Hetzner firewall. Tracked separately in Dev - Requests; not a blocker for these procedures.