Cloud operations guide

This guide is for people who run a StrongLinux cloud already deployed with strongctl (Server edition). Installation is covered in the installation guide, and version updates in the upgrade guide.

Every command runs as root on the host where strongctl was initialized (the first controller). The cloud configuration lives in /etc/strongctl/.

1. Everyday commands

To use the openstack client as administrator:

source /root/admin-openrc.sh
openstack server list --all-projects

2. Cloud health

strongctl verify checks, node by node:

• the systemd units of each role;
• the API ports and the TLS certificate of each one;
• token issuance and the catalog;
• compute services, network agents, volume and orchestration services;
• a secret cycle in Barbican (store, read, delete);
• with high availability, Galera (Synced), the RabbitMQ nodes, the OVN RAFT servers and the VIP on exactly one controller.

Run verify after any change and before an upgrade (which requires a green verify).

Published APIs

HAProxy publishes the APIs with TLS on the controller address (or on the VIP, with HA):

Behind HAProxy, each API listens only on the internal address (127.0.0.1 on an all-in-one). Nova metadata (8775) and the noVNC console (6080) are internal.

Logs

3. Credentials and secrets

• The vault /etc/strongctl/passwords.yml is encrypted with ansible-vault. Its password is in /etc/strongctl/.vault-pass (mode 0600).
• strongctl secrets show <name> shows a secret. If the name does not exist, the command lists the valid names.
• Important secrets: keystone_admin (the admin user), barbican_kek (master key for Barbican's secrets), strongstack_secret_key.
• Keep a copy of /etc/strongctl/ outside the cloud (section 5). Without barbican_kek, the secrets stored in Barbican cannot be recovered.

4. Adding and maintaining nodes

Add a compute or storage node

1. Install the node from the ISO (Compute or Storage option) or with the stronglinux-1.0-openstack-cloud-amd64.qcow2 image.
2. Authorize the controller's root SSH key on the new node (/root/.ssh/authorized_keys).
3. Add the node to /etc/strongctl/inventory.yml: the address under all.hosts and the name in the role's group.

   all:
     hosts:
       cmp3: {ansible_host: 10.0.0.23, mgmt_ip: 10.0.0.23}
     children:
       compute:
         hosts:
           cmp3:

4. Prepare the base and deploy:

   strongctl bootstrap
   strongctl deploy
   strongctl verify

Use the full deploy, without --role: it only changes the new node, but it also runs the compute registration in Nova (nova-manage cell_v2 discover_hosts), which lives on the first controller and does not run with --role compute.

Take a compute node out of service for maintenance

openstack compute service set --disable --disable-reason "maintenance" <host> nova-compute
openstack server list --all-projects --host <host>
openstack server migrate --live-migration <vm>     # or shut down the host's VMs

After the maintenance: openstack compute service set --enable <host> nova-compute.

Restarting a controller with high availability

Restart one controller at a time and check with strongctl verify before moving to the next. Galera, RabbitMQ and OVN rejoin the cluster on their own. If all three went down at the same time, follow the Galera recovery in the high availability guide.

5. Backups

strongctl does not make automatic backups. Keep, outside the cloud:

The services' configuration files (/etc/<service>/) do not need a copy: strongctl deploy regenerates them all from /etc/strongctl/.

To restore on a new controller: install the system, restore /etc/strongctl/ and the Keystone keys, run strongctl bootstrap, import the databases (zcat databases.sql.gz | mariadb) and run strongctl deploy and strongctl verify.

6. Token key rotation (fernet)

Keystone tokens are encrypted with fernet keys in /etc/keystone/fernet-keys/. Rotate the keys periodically, for example once a week. On the first controller:

runuser -u keystone -- keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
strongctl deploy --role controller      # with HA: copies the new keys to the other controllers

Keystone keeps 3 active keys, so tokens already issued remain valid. With a single controller, the first command is enough.

7. TLS certificates

• The API certificate is valid for 825 days and the internal CA for 10 years. To see the expiry date:

  openssl x509 -enddate -noout -in /etc/strongctl/tls/api.crt

• Renew the internal certificate: delete /etc/strongctl/tls/api.crt and run strongctl deploy.
• Use your own certificate: put api.crt (full chain), api.key and ca.crt in /etc/strongctl/tls/, without ca.key, and run strongctl deploy.
• Extra names in the certificate (for example a public DNS name): api_tls_names in globals.yml.

8. Day-to-day security

• Updates: unattended-upgrades applies Ubuntu's security fixes. StrongLinux's arrive through the stronglinux-1.0-security channel; on a cloud, apply them with strongctl upgrade.
• AppArmor: the OpenStack daemons run confined (enforce mode). To see what was denied:

  journalctl -k | grep 'apparmor="DENIED"'

  Local adjustments go in /etc/apparmor.d/local/stronglinux.<unit>, followed by apparmor_parser -r /etc/apparmor.d/stronglinux.<unit>. To diagnose, set apparmor_mode: complain in globals.yml and run strongctl deploy (logs only, does not block).

• Integrity (AIDE): compares the files every day; the database is renewed on every strongctl deploy. To check right away: aide --check -c /etc/aide/aide.conf.
• Account locked after 5 wrong passwords (15 min): faillock --user <name> --reset.
• SSH: after installing the keys, set PasswordAuthentication no.

9. Common problems

If the problem persists, open an issue with the output of strongctl verify and the service logs (without passwords). See the support page.