Backups and health
Delegaris remembers things that matter beyond its own settings. When someone keeps a machine on, Delegaris takes the schedule tag off and keeps its value until it puts it back. Losing the database while tags are off would strand those values. This page covers the nightly backup, restoring, the health check and the alerts.
What is stored, and where
With a single copy of the service, everything is in one SQLite file in the data volume: /data/delegaris.db (with the Linux package, /var/lib/delegaris/delegaris.db). That includes accounts and their saved keys, access rules, friendly names, keep-ons in progress, the tag values taken off resources, sessions and the activity log.
With PostgreSQL (DATABASE_URL), everything is in that database instead, and the built-in backup does not apply. Use the database's own backups, such as automated snapshots on Amazon RDS. See High availability.
The nightly backup
- A consistent copy of the database is written to
/data/backups(/var/lib/delegaris/backupswith the Linux package) each night at 02:00, in the time zone set under Settings > Schedules. It is taken while the service is running, using SQLite's own backup, so nothing has to stop. - Files are named
delegaris-YYYYMMDD-HHMMSS.db. The newest 14 are kept; older ones are deleted. BACKUP_HOURandBACKUP_KEEPchange the hour and how many are kept.BACKUP_ENABLED=falseturns it off.
Settings > Health and backups shows the last backup (when, how big, and where it went) and has a Back up now button.
Copies off the machine
A backup on the same disk does not help if the disk is lost. Set BACKUP_S3_BUCKET (and optionally BACKUP_S3_PREFIX, delegaris/ by default) and each backup is also uploaded to S3, with the service's own AWS identity. That identity then needs s3:PutObject on the bucket; iam/hub-role.yaml adds it when you give the bucket as BackupBucketName.
If the upload fails, the local copy still stands, and the failure shows under Settings > Health and backups and as an alert.
You can also copy the backups off the host yourself. With Docker:
docker compose cp delegaris:/data/backups ./delegaris-backups
With the Linux package, they are ordinary files in /var/lib/delegaris/backups, readable by root:
sudo cp -r /var/lib/delegaris/backups ./delegaris-backups
Restoring a backup
Stored secrets (account keys, the directory password and so on) are encrypted with a key that is not in the database: ENCRYPTION_KEY (with the Linux package, in /etc/delegaris/delegaris.env), or /data/encryption.key. A restore onto a new host needs the same key, or those secrets have to be entered again.
With Docker
-
Stop the container:
docker compose stop delegaris -
Copy the backup you want over
/data/delegaris.db. This runs a throwaway container from the same image and volume:docker compose run --rm --no-deps delegaris sh -c \'cp /data/backups/delegaris-20261112-020000.db /data/delegaris.db && rm -f /data/delegaris.db-wal /data/delegaris.db-shm'Removing the
-waland-shmfiles, if there are any, stops SQLite applying changes from the old database on top of the restored one. -
Start it again:
docker compose start delegaris
With the Linux package
Stop the service, copy the backup you want over the database (removing any -wal and -shm files), and start it again:
sudo systemctl stop delegaris
sudo -u delegaris cp /var/lib/delegaris/backups/delegaris-20261112-020000.db /var/lib/delegaris/delegaris.db
sudo rm -f /var/lib/delegaris/delegaris.db-wal /var/lib/delegaris/delegaris.db-shm
sudo systemctl start delegaris
Onto a new host
To restore onto a new host, set up Delegaris as usual with the same settings (.env, or /etc/delegaris/delegaris.env) and the same encryption key, then restore as above before anyone uses it.
Everything after the backup was taken is lost, including tag values taken off during that time. After a restore, check Activity and the resources that were being kept on.
Settings > Health and backups
This tab shows what the service checks about itself:
- A summary: either Nothing is wrong (the service can sign in to AWS and no timed stop is overdue), or a list of the problems found.
- Timed stops: when the background loop last ran and how often it runs.
- Last backup: when, how big, and whether it also went to S3; or why it failed.
- Copies off the machine: the S3 bucket backups are uploaded to, if any.
- Slack buttons: whether reminders carry +1 hour buttons, and whether the connection to Slack is up.
Check again refreshes it. The same tab holds the activity log export to S3.
Alerts
The service watches the things that must not fail quietly, and posts an alert when one does:
| Problem | When |
|---|---|
| The service cannot sign in to AWS | Only when something relies on its own identity: accounts reached through a role or the service's credentials, or backups to S3. Checked every few minutes. |
| Keep-ons are past their time and not finished | A keep-on is more than five minutes past its end time. Activity and the container log say why. |
| The last backup failed, or could not be uploaded to S3 | After a failed backup. |
| No backup for a day and a half | When the last backup is more than 36 hours old. |
An alert is posted when the problem starts, repeated every six hours while it lasts, and followed by a line when it clears. It goes to the Alerts channel under Settings > Notifications (or the Slack fallback channel if none is set), and to Microsoft Teams and the webhook if they are set up. Without any of those, the problems are logged and shown under Settings > Health and backups.
The health check: /healthz
GET /healthz needs no sign-in. Point your monitoring, load balancer or container platform at it.
| Answer | Meaning |
|---|---|
200 with {"status": "ok", "version": "0.9.0", "node": "…", "role": "active"} | All is well. role is active for the copy doing the timed work, standby for the others. |
503 with "status": "stalled" | The background loop that stops machines on time has not run for a while (five poll intervals, and at least two minutes). Nothing inside the service can raise the alarm about its own stopped clock, so this is how it says so. |
503 with "status": "read-only" | With PostgreSQL in a primary and standby pair: this copy is connected to the read-only standby. |
| An error | The database cannot be reached. |
The image has a Docker health check that calls /healthz every 30 seconds; after three failures in a row Docker marks the container unhealthy. Docker does not restart an unhealthy container by itself, so have whatever watches your containers act on it.
docker compose ps
curl -s http://127.0.0.1:8080/healthz
Logs
The container logs to standard output: docker compose logs delegaris. With the Linux package, the service logs to the systemd journal: delegaris logs (the same as journalctl -u delegaris -f). Besides startup lines and warnings, every entry in Activity is also written to the log as one line of JSON, so a log collector receives the audit trail as it happens. LOG_LEVEL sets how much is logged.