Multi-node cloud with high availability (strongctl)

With more than one controller in the inventory, strongctl builds a highly available control plane. No extra option is needed: the ha variable (in group_vars/all.yml) becomes true on its own.

Example: 3 controllers, 2 computes

strongctl init --vip 10.0.0.10 \
  --controller ctl1=10.0.0.11 --controller ctl2=10.0.0.12 --controller ctl3=10.0.0.13 \
  --compute cmp1=10.0.0.21 --network cmp1=10.0.0.21 \
  --compute cmp2=10.0.0.22 --storage cmp2=10.0.0.22
strongctl bootstrap && strongctl deploy && strongctl verify

strongctl runs on one of the controllers and reaches the nodes over SSH as root (the controller's root key in authorized_keys). --vip is a free IP on the management network: it is the public address of the APIs (controller_address).

What runs on each controller

Component	How it is made highly available
API VIP	Keepalived (unicast VRRP). The VIP stays on the highest-priority controller with a live HAProxy and moves within a few seconds if HAProxy or the node fails.
HAProxy	On all 3 controllers, listening on the VIP (`ip_nonlocal_bind`). Terminates TLS and balances the APIs across the 3.
APIs (uWSGI)	On all 3 controllers, listening on each one's management address.
MariaDB	Galera cluster on the 3 controllers. Services use VIP:3306, where HAProxy sends everything to a single node (the others stand by) to avoid write conflicts.
RabbitMQ	Cluster on the 3 controllers (shared cookie, `pause_minority`). Services get all 3 nodes in `transport_url` and use quorum queues (`rabbit_quorum_queue`).
memcached	On all 3 controllers. Services get the list of the 3.
OVN NB/SB	RAFT cluster on the 3 controllers. `ovn-northd` runs on all of them: one active, the others on standby. Neutron and the chassis get the list of the 3.
Keystone	On all 3 controllers, with the same fernet and credential keys (copied from the first).
Horizon	On all 3 controllers, with the same `SECRET_KEY` (a session works on any of them).
Glance	On all 3 controllers, but HAProxy sends everything to the first and keeps the others as backup, because the `file` store is local. With shared storage (NFS, Ceph), set `glance_shared_store: true` in globals.yml to use all three.
Metadata and noVNC	Listen on every controller and answer on the VIP wherever it is.

Database tasks (db sync, Nova cells, Glance metadefs, Keystone bootstrap) run only once, on the first controller.

Checks

strongctl verify also checks, on each controller:

the Galera cluster size and whether the node is Synced;
the RabbitMQ nodes;
the NB and SB RAFT servers;
that the VIP is on exactly one controller.

Recovery

A controller goes down and comes back: MariaDB rejoins Galera (IST/SST),

RabbitMQ rejoins the cluster and OVN rejoins RAFT on their own. Check with strongctl verify.

All three controllers went down: Galera does not start on its own. On the

node with safe_to_bootstrap: 1 in /var/lib/mysql/grastate.dat (the last one to stop), run galera_new_cluster; on the others, systemctl start mariadb.

MariaDB configuration change: the deploy restarts controllers 2 and 3 one

at a time. The first one needs a manual restart (systemctl restart mariadb), also one at a time.

Lab test

tests/lxd/run-multinode.sh builds the cloud above in 5 LXD containers and:

deploys and checks idempotency;
creates a VM on each compute node and tests the floating IP;
shuts down the VIP owner and checks that the cloud keeps working (token, lists, a new VM);
powers the controller back on and checks that it rejoins the cluster;
runs Tempest smoke.

Results go to out/multinode/.