High availability
One container is enough for most estates, but it is a single point of failure. If the server it runs on goes down at 9 pm, nothing puts the schedule tag back on the machine someone kept on until 10 pm, and the scheduler never stops it. To survive that, run two or more copies of Delegaris against a shared PostgreSQL database.
How it works
- Every copy keeps everything in the same PostgreSQL database: accounts, rules, sessions, keep-ons, bookings, approvals and the activity log. A session started on one copy works on the others, so a load balancer needs no stickiness.
- Every copy serves the web interface. Put them behind a load balancer.
- One copy at a time does the timed work: stopping machines at the end of a keep-on, sending reminders, running group starts and stops, opening booked windows, expiring approval requests, alerts, the weekly summary, the activity log export and price look-ups. It holds a lease in the database and renews it on every pass of its background loop (every 30 seconds).
- If the holder stops renewing (its server went down, the container was killed), another copy takes the lease once it has expired, after
LEASE_SECONDS(90 by default), and carries on from the database. The handover takes at most that long plus one poll interval. Nothing is lost: work that fell due in between is done on the new holder's first pass. - A copy that shuts down cleanly lets go of the lease at once, so another can take over without waiting for it to expire.
/healthzreports each copy'srole:activefor the lease holder,standbyfor the others.
Two things are deliberately not shared between copies:
- Sign-in throttling is per copy. An attacker gets the allowance once per copy, which is still five failures per user name in five minutes on each.
- The inventory cache is per copy: each copy asks AWS for itself. If AWS API rate limits ever show, raise
INVENTORY_CACHE_SECONDSa little.
Setting it up
-
Create a PostgreSQL database and set this in the
.envof every copy:DATABASE_URL=postgresql://delegaris:password@db.example.internal:5432/delegarisTables are created on first start. Copies starting at the same moment take turns, so they do not both try.
-
Give every copy the same
.envotherwise.SECRET_KEYmay be left unset: the key is made once and kept in the database.ENCRYPTION_KEYmust be set, to the same value on every copy: each copy otherwise makes its own key in its own/data, and a copy whose key does not match the stored secrets refuses to start.NODE_NAMEis optional and only names the copy in logs and in/healthz;NODE_NAME=$(hostname)written at boot is a good choice. -
Put them behind a load balancer with
GET /healthzas the health check. SetPUBLIC_URLto the load balancer's address andTRUST_PROXY_HEADERS=true. -
Run the same version on every copy. Pin
DELEGARIS_TAG; see Upgrading.
With PostgreSQL, the nightly file backup does not apply. Use the database's own backups.
demo/docker-compose.cluster.yml in the release repository runs two copies and a PostgreSQL container against the simulated AWS. Stop one copy and watch the other's /healthz report active within a minute. See Try the demo.
A highly available deployment on AWS
The shape that survives the loss of an instance, an Availability Zone or the database server, using nothing exotic. The service side of this is built and the handover between copies has been exercised; a full build-out and test on AWS is on the roadmap.
| Part | What to create | Why |
|---|---|---|
| Database | RDS for PostgreSQL, Multi-AZ, the smallest instance class (db.t4g.micro is plenty), automated backups on, kept 7 to 14 days | The single place everything lives. Multi-AZ fails over by itself in about a minute; the copies reconnect. |
| Compute | Two EC2 instances in two Availability Zones, each running the container from the standard docker-compose.yml, in an Auto Scaling group with minimum and maximum of 2 | A lost instance is replaced; a lost Availability Zone leaves one copy running. A t3.small is enough. |
| Identity | The instance profile from iam/hub-role.yaml on both instances, with the metadata hop limit at 2 (see Docker on EC2) | Both copies assume the same DelegarisRole in the managed accounts. No keys anywhere. |
| Configuration | One .env for both, kept in SSM Parameter Store (SecureString) or Secrets Manager and written to the instance at boot by user data, with DATABASE_URL, PUBLIC_URL, the sign-in settings and NODE_NAME=$(hostname) | Both copies must agree on everything. The instance role may read the parameter. |
| Entry point | An Application Load Balancer across both Availability Zones, HTTPS with an ACM certificate, forwarding to port 8080 on both instances. Health check GET /healthz, healthy threshold 2, interval 15 seconds. PUBLIC_URL=https://<the address people use> and TRUST_PROXY_HEADERS=true | TLS ends at the load balancer. It stops sending to a copy that fails the health check. No stickiness is needed: sessions are in the database. |
| Network | The instances in private subnets, the RDS security group open only to the instances' security group, the load balancer in public subnets (or internal, for a corporate network over VPN or Direct Connect). Outbound to the AWS APIs, your domain controllers on 636, and slack.com | The service is only ever reached through the load balancer. |
| Backups | RDS automated backups. BACKUP_ENABLED is irrelevant with PostgreSQL. | Point-in-time restore of everything. |
If you use the hub role for this, remember the instance role also needs to read the SSM parameter or secret; hub-role.yaml does not include that.
How it behaves
- The load balancer sends people to either copy. One copy holds the lease and does the stopping and reminding; the other serves the web interface.
- If the active copy's instance dies, the load balancer stops routing to it within about 30 seconds, and the other copy takes the lease within
LEASE_SECONDS(90) plus one poll (30 seconds). The worst case is a gap of about two minutes, during which nothing is lost, only delayed. The Auto Scaling group then brings a replacement up, which joins as standby. - If the database fails over, both copies see a few failed requests and carry on.
- Rolling out a new version is one instance at a time: replace it, wait for
/healthzto answer, then do the other.
The same shape runs on ECS Fargate: a service with two tasks across two Availability Zones, the task role in place of the instance profile, and the same load balancer and RDS database. The container needs no changes.
Two servers and nothing else: an active/passive database
Everything above rests on the database living somewhere that does not die with a server. If there is no RDS and PostgreSQL has to run in Docker on the same two servers, it has to be a primary and standby pair. Otherwise losing server one loses the database, and server two has nothing to work from.
This is not the recommended route. The intended shape for high availability is a separate database, such as RDS Multi-AZ, with two or more copies of Delegaris in front of it, as above. The two-server form is described for completeness; plan and test it as a database operations task in its own right.
-
Server one runs PostgreSQL as the primary and a copy of Delegaris. Server two runs PostgreSQL as a hot standby, fed by streaming replication (
pg_basebackuponce, thenprimary_conninfo), and a copy of Delegaris. -
Both copies of Delegaris are given both database hosts:
DATABASE_URL=postgresql://user:pw@/delegaris?host=server-one:5432&host=server-two:5432&target_session_attrs=read-writeThe client tries them in turn and settles on the one that accepts writes, so after a failover nothing in Delegaris needs changing. A copy whose connection has landed on the standby answers
/healthzwith503("status": "read-only") until it finds the primary again, so a load balancer stops sending people to it. -
Who promotes the standby is the decision that makes or breaks this. Promotion by hand (
pg_ctl promoteon server two) is simple and safe but needs a person, which at 9 pm nobody is. Automatic promotion with two servers needs a referee, or a network hiccup between them leaves both believing they are primary, and the two databases drift apart. The standard tool is pg_auto_failover: its monitor runs as a third small PostgreSQL. On two servers, put it on server two; losing server one is then handled, and losing server two leaves server one running, unmonitored, which is fine. Patroni and repmgr are the alternatives and want a third vote too. -
After a failover, the old primary comes back as a stale copy and must be re-attached as the new standby (
pg_rewind, or a freshpg_basebackup). pg_auto_failover does this for you.
This is the one place where RDS Multi-AZ, at about the price of a small instance, earns its keep: it is exactly this pair, with the referee, run by AWS. If the two-server form is the one you want, the demo's docker-compose.cluster.yml is the place to start: swap its single postgres service for a primary and a standby, and give both copies the two-host address.