Roles reference¶

All roles live in roles/. Each role has a defaults/main.yml with documented variables and a tasks/main.yml that does the work.

Overview¶

Role details¶

access_users¶

Creates OS user accounts, adds SSH public keys, and configures sudo.

Users are defined in two ways that are merged together: 1. access_users – a flat list (defaults to [pt] from group_vars/all/common.yml) 2. system.users – per-host structured users

# group_vars/all/common.yml
access_users:
  - name: pt
    groups: [sudo, docker]
    key_files:
      - keys/pt.pub

# host_vars/bkdk-web-01/main.yml
system:
  users:
    - name: barberklingen
      sudo: true
      ssh:
        generation: true     # generate ed25519 key pair on server
        authorized_key: true # self-authorize the generated key
        connect: true        # add to SSH AllowUsers

Key files are resolved relative to the playbook root (the repo root). Store public keys in keys/.

The role is additive for declared users — it never removes an account, key, or group membership. The one exception is access_users_denied: names on that list are refused, and any access they still hold is stripped. See Revoked access.

auditd¶

Installs auditd and deploys a baseline ruleset that logs changes to: - Identity files (/etc/passwd, /etc/shadow, /etc/sudoers) - SSH config - Cron directories - Package management executables - Systemd and Docker binaries

auditd_enabled: true   # set to false in sandbox

# Append host-specific rules
auditd_extra_rules:
  - "-w /etc/mysql/ -p wa -k mysql"

cadvisor¶

Runs cAdvisor as a Docker container for exposing container resource metrics.

cadvisor_enabled: true
cadvisor_listen_port: 8080
cadvisor_image: "gcr.io/cadvisor/cadvisor:v0.49.1"

Waits for the Docker daemon to be ready before starting the container.

cloudflare¶

See the dedicated Cloudflare guide for full documentation.

Key defaults:

cloudflared_remote_config: true
cloudflared_create_tunnel: true
cloudflared_dns_proxied: true
cloudflare_application_session_duration: "24h"

common_baseline¶

Runs first on every host to establish a consistent baseline: - Removes any legacy Docker APT repository definitions that would conflict with the current setup. - Updates the APT cache. - Installs base packages: ca-certificates, curl, git, ufw, vim. - Sets the server timezone.

timezone: Etc/UTC
common_packages: []   # append additional packages here

cronjobs¶

Deploys cron shell scripts from cronjobs/ to /opt/cronjobs/<dir>/ and installs crontab entries for the site user.

Cron jobs are defined globally in group_vars/all/cronjobs.yml and enabled per-site in host_vars:

# group_vars/all/cronjobs.yml – job definitions
cronjobs:
  - name: wp-cron
    path: "/opt/cronjobs/{{ site.cronjob.dir }}/wp-cron.sh"
    timing: "*/5 * * * *"

# host_vars/bkdk-web-01/main.yml – enable per site
sites:
  - name: barberklingen
    cronjob:
      dir: barberklingen-dk
      enable:
        - name: wp-cron
        - name: as-global

deployment¶

Creates deployment OS users with generated SSH key pairs and restricted sudo (limited to specific commands and paths). Used by CI/CD pipelines to deploy application code.

Users are defined in two lists that are merged at run time:

• system_users_base – defined in group_vars/all/deployment.yml, always includes the deploy user
• system_users_extra – defined per-environment (e.g. group_vars/web/main.yml) for additional users such as branch deployments

# group_vars/all/deployment.yml
system_users_base:
  - name: deploy
    groups: [sudo]
    ssh:
      authorized_keys: true
      create_keys: true

# inventories/prod/group_vars/web/main.yml
system_users_extra:
  - name: "{{ system_user_branch_name | default('') }}"
    pwd: "{{ system_user_branch_password | default('') }}"
    groups: [sudo]
    ssh:
      create_keys: true
      authorized_keys: true

See SSH management for full details on managing extra users.

docker_engine¶

Installs Docker CE from the official Docker APT repository. Configures the Docker daemon and ensures the service is running and enabled.

docker_compose_plugin¶

Installs the Docker Compose V2 plugin (docker compose). Requires docker_engine to have run first.

fail2ban¶

Installs Fail2ban to block SSH brute-force attempts.

fail2ban_bantime: "1h"
fail2ban_findtime: "10m"
fail2ban_maxretry: 5

firewall_ufw¶

Configures UFW with a deny-all inbound, allow-all outbound baseline. Opens specific TCP ports and source CIDRs as needed.

ufw_default_incoming: deny
ufw_default_outgoing: allow
ufw_allow_tcp_ports: []   # e.g. [80, 443]

When zt_lockdown_enabled: true, the role allows no inbound ports (SSH is served via Cloudflare Tunnel only).

IP allowlists from group_vars/all/allowlist.yml are used to grant MySQL access from office IPs, Plecto, and Supermetrics CIDRs.

journald_limits¶

Configures systemd-journald to cap disk usage and retention:

journald_system_max_use: "500M"
journald_max_retention_sec: "7day"

Restarts journald if the config changes.

logrotate_custom¶

Deploys custom logrotate configurations for application log paths defined in sites[*].logrotate.system_logs_glob_paths.

mariadb¶

Installs MariaDB, hardens the installation (removes anonymous users, test databases), and manages databases and users declared in host variables.

Authentication for local root operations uses login_unix_socket (compatible with MariaDB's plugin-based auth). No root password prompt needed.

mariadb_bind_address: "0.0.0.0"   # set in group_vars/all/db.yml
mariadb_server_id: 1               # must be unique across all DB servers (used for replication)

mariadb_databases:
  - name: myapp_db
    collation: utf8mb4_unicode_ci
    encoding: utf8mb4

mariadb_users:
  - name: myapp
    host: "10.0.0.5"
    password: "{{ vault_myapp_db_password }}"
    priv: "myapp_db.*:ALL"

# Secrets in vault:
vault_mariadb_root_password: "..."

mariadb_binlog¶

Enables and configures binary logging for point-in-time recovery.

mariadb_binlog_enabled: true
mariadb_binlog_retention_days: 7
mariadb_binlog_format: "ROW"

netdata¶

Installs the Netdata monitoring agent for real-time system metrics. Enable the server integration per-host:

# host_vars/<hostname>/main.yml
system:
  netdata:
    active: true   # registers this server in the Netdata dashboard

sqlbak¶

Registers the server with SqlBak for automated database backups.

• Downloads and runs the SqlBak install script (verified by SHA256).
• Registers the server using sqlbak_server_key.
• Configures a MariaDB connection for automated backups.
• Uses no_log: true for registration commands to avoid leaking credentials.

# Secrets in group_vars/prod/db.vault.yml:
sqlbak_db_user: "sqlbak"
sqlbak_db_password: "..."
sqlbak_install_script_url: "https://..."
sqlbak_install_script_sha256: "..."

# What mysqldump needs to produce a complete dump - see below
sqlbak_db_privs: "*.*:SELECT,SHOW VIEW,TRIGGER,EVENT,PROCESS,RELOAD,LOCK TABLES,REPLICATION CLIENT"

Idempotency: checks sqlbak --info to determine if the server is already registered before re-registering.

EVENT and TRIGGER are not optional even on a database that uses neither. mysqldump issues SHOW EVENTS regardless and aborts the entire dump on the resulting error 1044, which is how the incremental backups failed on 2026-08-25.

Setting sqlbak_db_user: "" skips the four database tasks — the agent install and server registration still run. That is how tools-server-01 is configured: the host has no database to back up, only files. It is also a workaround, not a design: the role's mysql_user task has no login_unix_socket, unlike the equivalent task in the mariadb role, so it fails with error 1698 on a host where root authenticates through the socket.

restore_verify¶

Downloads the newest SqlBak backup for each production database from S3, restores it into a bckp_test_* scratch database on tools-server-02, checks the result and deletes everything again. Reports to Sentry Cron Monitors, one per database.

The cron monitor is the alarm; it names the database but carries no free text. The reason for a failure goes to Sentry Logs over the OTLP endpoint, which is derived from the same DSN and shares a trace id with the check-in. Both use vault_restore_verify_sentry_public_key — there is no second secret.

restore-verify-selftest.sh is deployed alongside the job and proves the alarm actually fires: it breaks one expectation at a time and asserts that the failure reaches Sentry. It is never run automatically.

See the dedicated Backup verification guide for the full picture, including how to add a database and how to run it by hand.

# host_vars/tools-server-02/main.yml
restore_verify_enabled: true            # default false; only host in the allow list
restore_verify_s3_bucket: "..."

restore_verify_targets:
  - name: bkdk
    source_host: bk-db-01
    s3_prefix: "db/daily/bkdk/"
    target_database: bckp_test_bkdk     # must match ^bckp_test_

# Secrets in vault:
vault_restore_verify_aws_access_key_id: "..."
vault_restore_verify_aws_secret_access_key: "..."
vault_restore_verify_archive_password: "..."
vault_restore_verify_mysql_password: "..."
vault_restore_verify_sentry_cron_base: "..."
vault_restore_verify_sentry_public_key: "..."

The role hard-fails if enabled on a host outside restore_verify_allowed_hosts, if any target_database is not a bckp_test_* name, or if binary logging is enabled on the host. The MariaDB user it creates is granted privileges only on the declared scratch databases.

Three deliberate deviations from the rest of the repository:

• It uses a systemd timer rather than the cronjobs role, which is site-driven and runs as the site user. A root-owned system job needs a runtime limit, PrivateTmp and journal logging.
• The runtime limit is TimeoutStartSec, not RuntimeMaxSec. systemd ignores RuntimeMaxSec for Type=oneshot and logs that it has no effect, so the earlier TimeoutStartSec=infinity left the job with no limit at all. Note it bounds the whole run, not each database.
• It installs the AWS CLI from a pinned zip verified against AWS' PGP signature, because AWS publishes signatures rather than checksums. Ubuntu's awscli package is v1 and is not used.

The archive password comes from vault, lands in a 0600 file owned by root and is passed to 7z over stdin, so it never reaches a process list or the journal.

The archives turn out to be ZipCrypto, not AES-256, despite the setting in the SqlBak UI. ZipCrypto is broken against a known-plaintext attack and a SQL dump is a near-ideal plaintext, so the password is an obstacle rather than protection — confidentiality rests on the bucket policy and the IAM keys. See docs/backup-verification.md.

seven_zip¶

Installs 7-Zip and verifies the binary reports AES support, which SqlBak needs to write encrypted archives and restore_verify needs to read them. 7z is used for .zip as well as .7z, because Ubuntu's unzip cannot read AES-encrypted zip files.

The assertion is about the binary's capability. It says nothing about which cipher SqlBak actually uses — in practice that is ZipCrypto, which is a separate finding recorded in docs/backup-verification.md.

seven_zip_enabled: false        # set true per group, e.g. group_vars/tools/main.yml
seven_zip_package: "7zip"
seven_zip_command: "7z"

ssh_hardening¶

Writes a hardened sshd_config: - PermitRootLogin no - PasswordAuthentication no - PubkeyAuthentication yes - AllowUsers populated from ssh_allow_users

ssh_port: 22
ssh_permit_root_login: "no"
ssh_password_authentication: "no"
ssh_pubkey_authentication: "yes"

AllowUsers is computed dynamically from base_ssh_allow_users, access_users, system.users (with ssh.connect: true), and sites[*].user.

SSH tunnels (port forwarding)¶

AllowTcpForwarding no applies to every host, so SSH tunnels are blocked by default. A client that tries one gets administratively prohibited: open failed.

To allow tunnels for named users on a single host, set both variables in that host's host_vars:

# inventories/stage/host_vars/web-01/main.yml
ssh_tcp_forwarding_users:
  - pt
ssh_tcp_forwarding_permitopen:
  - localhost:3320

This renders a Match User block at the bottom of sshd_config that reopens forwarding for those users and only towards the listed host:port destinations. Everyone else, and every other host, keeps AllowTcpForwarding no.

The exception is AllowTcpForwarding local, so it covers ssh -L only. Remote listeners (ssh -R) stay blocked – PermitOpen does not restrict those, so allowing them would let the user expose arbitrary services on the server.

Setting ssh_tcp_forwarding_users without ssh_tcp_forwarding_permitopen fails the play – a tunnel must always name where it may go. Names in access_users_denied are refused here too.

unattended_upgrades¶

Enables automatic security updates via unattended-upgrades. Only security patches are applied automatically; major package upgrades require manual intervention.

verify_cloudflared¶

Asserts that the cloudflared systemd service is active and healthy before the lockdown.yml playbook proceeds. Fails with a clear error if the tunnel is not running.

cloudflared_required: true   # set automatically by lockdown.yml