IT Contingency Plan¶
| Owner | Patrick Tolvstein |
| Version | 1.1 |
| Created | 2026-06-12 |
| Last reviewed | 2026-06-16 |
| Next review | 2027-06-16 |
Use this when a critical system or service is unavailable for an extended period and normal operations cannot continue. This plan covers workarounds and minimum viable operations - not full recovery ( see Disaster Recovery for that).
Critical systems¶
| System | Impact if unavailable | Workaround | Plan |
|---|---|---|---|
| WooCommerce sites | No orders, no revenue | Maintenance page + manual order handling | - |
| Email (SMTP) | No emails can be sent internally | Switch SMTP provider to Mailgun (requires DNS updates) | - |
| Klaviyo (Mails) | No transactional emails | Enable transactional emails in WooCommerce | - |
| Payment provider | No checkout | Disable affected provider, and switch to backup provider if available | - |
| Satis (packagist.subscribed.build) | Cannot deploy packages | Use direct Git dependencies temporarily | - |
| Server (production) | Full site outage | Failover to backup server or static maintenance page (Cloudflare) | Disaster Recovery |
| GitHub | No deployments, no CI | Deploy from a local machine over the Cloudflare Access SSH tunnel | - |
| Cloudflare Tunnel (cloudflared) | No SSH access to locked-down hosts; web hosts also stop serving customers | Restore access via Hetzner Console, then route around the tunnel | Zero Trust Resilience |
| Cloudflare (account or edge outage) | DNS, CDN, WAF and all admin access down | DNS is hosted at Cloudflare and cannot be repointed. Accepted risk - communicate and wait | Known limitation |
| VPN | No access to restricted systems | Remove VPN policies and replace with SSO policy instead | - |
Declaring a contingency¶
A contingency is declared when:
- A critical system has been unavailable for more than 2 hours with no clear resolution timeline
- Normal operations (orders, deployments, communication) cannot continue
- The incident cannot be resolved with standard troubleshooting
The Incident Commander declares the contingency and notifies the team.
Communication during contingency¶
- Activate out-of-band communication (Signal / phone) if email or Slack is affected
- Update internal status every hour
- If customer-facing: put up a status page or maintenance notice
- Keep a running log of decisions and actions taken
Returning to normal operations¶
Before lifting the contingency:
- Root cause is identified and resolved
- System integrity has been verified
- Data consistency check completed (orders, users, payments)
- Team notified that normal operations have resumed
- Incident log updated with resolution details