Skip to content

Disaster Recovery
Owner Patrick Tolvstein
Version 1.0
Created 2025-08-24
Last reviewed 2026-06-16
Next review 2027-06-16

Use this when there is data loss, server failure, or a catastrophic outage that cannot be resolved by standard troubleshooting or the contingency plan. The goal is to restore the system to a known-good state as quickly and safely as possible.

Recovery objectives

Metric Target
RTO (Recovery Time Objective) < 4 hours for P1 outages
RPO (Recovery Point Objective) < 24 hours (last successful backup)

Backup inventory

Asset Backup method Frequency Location Retention
Database (MariaDB) Automated dump Daily SQLBak or Storage Box 30 days
WordPress uploads File sync Daily SQLBak or Storage Box 30 days
Server configuration Ansible (IaC) On change GitHub Indefinite
DNS records Cloudflare export On change Cloudflare -

Info

Ansible playbooks in subscribed-aps/tools-server-configuration are the source of truth for server state. A server can be fully reprovisioned from scratch using them.

Recovery scenarios

Scenario 1 - Database corruption or loss

  1. Stop the application to prevent further writes
  2. Identify the last clean backup
  3. Restore the database: 
    mysql -u root -p [database_name] < backup_YYYY-MM-DD.sql
    Or use SQLBak for restoring the database directly from their interface.
  4. Verify data integrity (spot-check recent orders, users - check for IDs, dates etc.)
  5. Restart the application and monitor

Scenario 2 - Server failure (unrecoverable)

  1. Provision a new server using Ansible: 
    ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.yml --limit new-server
ansible-playbook playbooks/site.yml -i inventory/hosts.yml --limit new-server
  2. Restore database from latest backup (Not relevant if the failure is only related to edge servers)
  3. Restore WordPress uploads (Not relevant if the failure is only on alpha servers)
  4. Update DNS to point to new server IP (Cloudflare)
  5. Verify SSL certificates are active
  6. Run smoke tests (place test order, check all pages)

Scenario 3 - Ransomware / total compromise

  1. Isolate all affected systems immediately (see Incident Response)
  2. Do not pay the ransom before consulting legal and cyber insurance
  3. Provision clean servers from scratch using Ansible
  4. Restore only from backups predating the compromise
  5. Reset all credentials (database passwords, API keys, SSH keys, admin accounts)
  6. Rotate all secrets and tokens before going live
  7. Report to Data Protection Authority if personal data was affected (72-hour GDPR deadline)

Post-recovery checklist

  • All sites loading correctly
  • Checkout and payment flow tested end-to-end
  • Transactional emails sending
  • Admin access confirmed for all team members
  • No data loss beyond the RPO window
  • Incident log updated
  • Post-mortem scheduled within 5 business days