Skip to content

Disaster Recovery
Owner Patrick Tolvstein
Version 1.0
Created 2025-08-24
Last reviewed 2026-06-16
Next review 2027-06-16

Use this when there is data loss, server failure, or a catastrophic outage that cannot be resolved by standard troubleshooting or the contingency plan. The goal is to restore the system to a known-good state as quickly and safely as possible.

Recovery objectives

Metric Target
RTO (Recovery Time Objective) < 4 hours for P1 outages
RPO (Recovery Point Objective) < 24 hours (last successful backup)

Backup inventory

Asset Backup method Frequency Location Retention
Database (MariaDB) Automated dump Daily SQLBak or Storage Box 30 days
WordPress uploads File sync Daily SQLBak or Storage Box 30 days
Server configuration Ansible (IaC) On change GitHub Indefinite
DNS records Cloudflare export On change Cloudflare -

Info

Ansible playbooks in subscribed-aps/tools-server-configuration are the source of truth for server state. A server can be fully reprovisioned from scratch using them.

Recovery scenarios

Scenario 1 - Database corruption or loss

  1. Stop the application to prevent further writes
  2. Identify the last clean backup
  3. Restore the database: 
    mysql -u root -p [database_name] < backup_YYYY-MM-DD.sql
    Or use SQLBak for restoring the database directly from their interface.
  4. Verify data integrity (spot-check recent orders, users - check for IDs, dates etc.)
  5. Restart the application and monitor

Scenario 2 - Server failure (unrecoverable)

  1. Provision a new server using Ansible - see New server for the full workflow: 
    ansible-playbook -i inventories/prod/hosts.yml bootstrap_root.yml --limit new-server
ansible-playbook -i inventories/prod/hosts.yml site.yml --limit new-server
  2. Restore database from latest backup (Not relevant if the failure is only related to edge servers)
  3. Restore WordPress uploads (Not relevant if the failure is only on alpha servers)
  4. Repoint DNS in Cloudflare. Locked-down hosts are not reached by an A record to the server IP - the record is a CNAME to <tunnel-uuid>.cfargotunnel.com. Bring the cloudflared connector up on the new host first, then point the record at the new tunnel. See Zero Trust Resilience for how the tunnel path fits together.
  5. Re-apply lockdown once the tunnel is verified healthy: 
    ansible-playbook -i inventories/prod/hosts.yml lockdown.yml --limit new-server
  6. Verify SSL certificates are active
  7. Run smoke tests (place test order, check all pages)

Scenario 3 - Ransomware / total compromise

  1. Isolate all affected systems immediately (see Incident Response)
  2. Do not pay the ransom before consulting legal and cyber insurance
  3. Provision clean servers from scratch using Ansible
  4. Deploy application code from Git via the CI/CD pipeline — do not restore the full deployments folder from a file backup, as it may contain the malicious code that caused the compromise. Restore only web/app/uploads/ and the database from backups predating the compromise. See Backups — Restoring a compromised site for the detailed procedure.
  5. Reset all credentials (database passwords, API keys, SSH keys, admin accounts)
  6. Rotate all secrets and tokens before going live
  7. Report to Data Protection Authority if personal data was affected (72-hour GDPR deadline)

Post-recovery checklist

  • All sites loading correctly
  • Checkout and payment flow tested end-to-end
  • Transactional emails sending
  • Admin access confirmed for all team members
  • No data loss beyond the RPO window
  • Incident log updated
  • Post-mortem scheduled within 5 business days