Playbooks¶

All playbooks live in the repo root. Use make or ansible-playbook directly.

Quick reference¶

The tool servers have their own inventory, inventories/tool-server/hosts.yml, rather than living in prod. make tool points at it directly, so ENV only matters if you override the inventory yourself.

Makefile usage¶

The Makefile defaults to the stage inventory. Override with ENV=prod.

# Ping all staging hosts
make ping

# Run web playbook on prod
make ENV=prod web

# Run db playbook on prod
make ENV=prod db

# Full site run on prod
make ENV=prod site

The Makefile includes VAULT ?= --ask-vault-pass. Because ansible.cfg already points to .ansible_vault_pass, you normally don't need vault prompts. Suppress the flag:

make ENV=prod VAULT="" web

Playbook details¶

site.yml – full site run¶

Imports both web.yml and db.yml. Use this to converge the entire environment.

ansible-playbook -i inventories/prod/hosts.yml site.yml

web.yml – web servers¶

Applies the complete web server configuration to the web group.

Roles applied (in order): common_baseline → access_users → deployment → journald_limits → logrotate_custom → unattended_upgrades → firewall_ufw → ssh_hardening → fail2ban → auditd → docker_engine → docker_compose_plugin → cadvisor → netdata → cloudflare → cronjobs

# All web servers
ansible-playbook -i inventories/prod/hosts.yml web.yml

# Single host
ansible-playbook -i inventories/prod/hosts.yml web.yml --limit bkdk-web-01

db.yml – database servers¶

Applies the complete database server configuration to the db group.

Roles applied (in order): common_baseline → access_users → journald_limits → logrotate_custom → unattended_upgrades → firewall_ufw → ssh_hardening → fail2ban → auditd → cloudflare → netdata → mariadb → mariadb_binlog → sqlbak

ansible-playbook -i inventories/prod/hosts.yml db.yml --limit bk-db-01

baseline.yml – hardening only¶

Runs security hardening roles on all hosts without app-specific configuration. Useful for mass updates to security policy.

Roles: common_baseline, journald_limits, logrotate_custom, unattended_upgrades, firewall_ufw, ssh_hardening, fail2ban, auditd

ansible-playbook -i inventories/prod/hosts.yml baseline.yml

bootstrap_root.yml – first-time root bootstrap¶

Used when you only have root SSH access to a fresh server. Creates the operator users from access_users, installs their public keys, configures sudo, and applies baseline hardening.

Runs as remote_user: root. Does not use become.

ansible-playbook -i inventories/prod/hosts.yml bootstrap_root.yml --limit <new-host>

See New server for the full first-time workflow.

bootstrap_web.yml – web bootstrap¶

Extends baseline.yml with Docker Engine, Docker Compose plugin, and cloudflared installation. Run this after bootstrap_root.yml on new web servers.

ansible-playbook -i inventories/prod/hosts.yml bootstrap_web.yml --limit <new-web-host>

bootstrap_db.yml – DB bootstrap¶

Extends baseline.yml with MariaDB, binary log configuration, SqlBak, and cloudflared. Run this after bootstrap_root.yml on new database servers.

ansible-playbook -i inventories/prod/hosts.yml bootstrap_db.yml --limit <new-db-host>

lockdown.yml – Zero Trust lockdown¶

Closes all inbound UFW ports and binds SSH to localhost. After this, the server is only reachable via Cloudflare Tunnel. Requires cloudflared to already be running (verified before applying).

ansible-playbook -i inventories/prod/hosts.yml lockdown.yml --limit <web-host>

Warning: If cloudflared is not healthy when this runs, the playbook will abort. Do not run this unless you have verified tunnel connectivity.

tool.yml – tool servers¶

Configures the tools group: Verdaccio (private npm registry), deployment tooling, and monitoring.

ansible-playbook -i inventories/prod/hosts.yml tool.yml

analytics.yml – analytics servers¶

Configures the analytics group: Docker-based analytics stack with standard hardening and Cloudflare integration.

ansible-playbook -i inventories/prod/hosts.yml analytics.yml

access.yml – operator accounts and SSH access¶

Runs only access_users and ssh_hardening. Use it when changing who may log in, instead of a full web.yml/db.yml run - the blast radius is limited to users, groups, authorized_keys, sudoers and sshd_config. It touches nothing to do with the firewall, Cloudflare, Docker or cron.

ansible-playbook -i inventories/prod/hosts.yml access.yml --check --diff
ansible-playbook -i inventories/prod/hosts.yml access.yml --diff

It renders the same sshd_config as the full playbooks, because zt_lockdown_enabled comes from the inventory rather than a play variable.

See Revoked access for what happens to a name on access_users_denied.

root_keys.yml – root's authorized_keys¶

access_users deliberately does not manage /root/.ssh/authorized_keys, so this playbook is the deliberate exception. It authorizes the operator key for root and can revoke an offboarded operator's key in the same pass, in that order - on some hosts the offboarded key is the only root key, and revoking it first would leave none.

# Authorize the operator key for root
ansible-playbook -i inventories/prod/hosts.yml root_keys.yml --check --diff

# Authorize it and revoke someone's key in one pass
ansible-playbook -i inventories/prod/hosts.yml root_keys.yml \
  -e revoke_pubkey_file=keys/olduser.pub --diff

revoke_pubkey_file has no default on purpose: a name baked into the playbook would make every later run revoke that person's key.

In --check the assert is skipped, because it cannot verify a key that has not been written yet. The dry run therefore shows the revoke as a change even on hosts where the operator key is still missing - that is expected, and the real run adds the key first.

Useful flags¶

Example – dry run with diffs on a single host:

ansible-playbook -i inventories/prod/hosts.yml web.yml --limit bkdk-web-01 --check --diff