Automatic restore verification

How a verification runs

The verifier is an agent with the verifier role on a separate machine. The stages run in order, the first one that fails stops the verification, and cleanup always runs:

• the snapshot exists in the repository, and the fingerprint in the job matches the hash in the snapshot’s tag: the server is not trusted;
• the work volume is mounted noexec and has enough space;
• restic check reading a part of the data;
• for databases: a sandbox with a container of the right PostgreSQL version, with no network;
• restore: restic restore --verify or pg_restore in the sandbox;
• the copy opens in its own application;
• the copy’s fingerprint, taken by the plugin’s code, matches the source’s fingerprint;
• the copy, the container and its volumes are removed.

Five levels and the restore time

The outcome is the highest level reached: storage integrity, restore, start, meaning of the data. At level five every stage has passed and the restore time is measured: the actual RTO.

Five outcomes, not two

Passed, failed, cannot verify, infrastructure failure and cancelled. A failed backup and a failure of the check itself are kept apart: a broken verifier does not look like a bad backup, and a failed restore does not look like a warning.

Verifier security

• The verifier refuses to verify a snapshot of its own machine, whatever the server sends.
• It has its own repository key: the source’s password is not moved to the verifier.
• Storage access is read-only, and every restic command runs without locking the repository.
• Restored files are never executed: the work volume is mounted noexec.
• The sandbox container has no network and limits on CPU, memory and processes, and the server password inside is random for each verification.
• Data goes from storage to the verifier, bypassing the server. Only the fingerprint, the snapshot id, jobs and reports pass through the server.

What you see in the console and in Telegram

• The last verified restore: the main metric of the overview, and a metric of its own for each source.
• A “Verifications” tab for a source, and a record of every verification: stages, durations, fingerprints side by side.
• Mismatches in words: what exactly differs between the source and the copy.
• A “no verifier” hint with ready commands when nothing can verify.
• A “not verified for long” threshold (1 to 90 days) and Telegram notifications about verifications.
• A printable report of a source’s verifications.

What is verified today

PostgreSQL (pg_dump) and file backups. Verification runs as a daily cycle on a schedule set in the settings, and on demand from the console. For databases the verifier needs Docker or Podman; we recommend rootless Podman. Other sources will arrive with their own plugins.