# index.html.md
sunbeam`).
```
Create the file `credentials.yaml` to define your cloud credentials:
```yaml
credentials:
sunbeam:
sunbeam-creds:
auth-type: userpass
username: $OS_USERNAME
password: $OS_PASSWORD
tenant-name: $OS_PROJECT_NAME
project-domain-name: $OS_USER_DOMAIN_NAME
user-domain-name: $OS_USER_DOMAIN_NAME
version: "$OS_AUTH_VERSION"
```
Like before, you can query for the value of each variable. Note that the
value for `version` is in double quotes.
Add your credentials to Juju:
```default
juju add-credential sunbeam --client -f credentials.yaml
```
Example output:
```default
Credential "sunbeam-creds" added locally for cloud "sunbeam".
```
## Create a Juju controller
The steps in this section are also performed on the bastion.
Create a Juju controller, here named `my-controller`:
```default
juju bootstrap sunbeam my-controller
```
End of example output:
```default
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 192.168.122.220 to verify accessibility...
Bootstrap complete, controller "my-controller" is now available
Controller machines are in the "controller" model
Now you can run
juju add-model
to create a new model to deploy workloads.
```
## Deploy an application
You can now use standard Juju practices to manage applications. See the
[Juju documentation](https://juju.is/docs/juju) for help with Juju.
Below, we’ll create a model and add the `ubuntu` application to it.
```default
juju add-model my-model
juju deploy ubuntu --base ubuntu@22.04
```
To inspect the model:
```default
juju status
```
Example output:
```default
Model Controller Cloud/Region Version SLA Timestamp
my-model my-controller sunbeam/RegionOne 3.4.2 unsupported 15:07:44Z
App Version Status Scale Charm Channel Rev Exposed Message
ubuntu 22.04 active 1 ubuntu latest/stable 24 no
Unit Workload Agent Machine Public address Ports Message
ubuntu/0* active idle 0 192.168.122.52
Machine State Address Inst id Base AZ Message
0 started 192.168.122.52 4c147f10-9f9e-449b-b58a-6b9534553e4a ubuntu@22.04 nova ACTIVE
```
Log out of the bastion in preparation for the next section:
```default
exit
```
## Verify the OpenStack server instances
On the client host, via the `openstack` CLI, you can see the OpenStack
server instances that correspond to the workload machine, the Juju controller,
and the bastion (respectively, from top to bottom, in the output below):
```default
openstack server list
+--------------------------------------+--------------------------+--------+-------------------------------------------+--------------------------------------------------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------------------+--------+-------------------------------------------+--------------------------------------------------------------+-----------+
| 4c147f10-9f9e-449b-b58a-6b9534553e4a | juju-08056b-my-model-0 | ACTIVE | demo-network=192.168.122.52 | auto-sync/ubuntu-jammy-22.04-amd64-server-20240319-disk1.img | m1.small |
| e0b7858f-4529-442e-8440-b8fde6819347 | juju-8cf50d-controller-0 | ACTIVE | demo-network=192.168.122.220 | auto-sync/ubuntu-jammy-22.04-amd64-server-20240319-disk1.img | m1.medium |
| ba8c4cfe-0e27-4471-9923-a7fbedf774c5 | bastion | ACTIVE | demo-network=10.20.30.188, 192.168.122.32 | ubuntu | m1.tiny |
+--------------------------------------+--------------------------+--------+-------------------------------------------+--------------------------------------------------------------+-----------+
```
# index.html.md
# Configuring DPDK
## Overview
Open vSwitch (OVS) can be configured to use the [DPDK](https://www.dpdk.org) (Data Plane Development
Kit) userspace datapath, achieving increased performance compared to the
standard OVS kernel datapath.
## Prerequisites
### Compatible network adapters and CPU architecture
Please consult the [DPDK supported hardware page](https://core.dpdk.org/supported/) to ensure that your CPU
architecture and network adapters are compatible with DPDK.
### Isolated CPU cores
Canonical Openstack users can specify the desired number of CPU cores that will
be allocated to DPDK. The cores will be distributed based on the NUMA location
of the network interfaces used with DPDK.
Use kernel command line parameters to preconfigure the necessary number of
isolated CPU cores and pay attention to the NUMA nodes:
```default
isolcpus=0-3,16-19
```
If hyperthreading is enabled, make sure to include the CPU siblings as well.
### VT-d / IO-MMU
IO virtualization (for example Intel VT-d or AMD-V) must be enabled in
the system BIOS and then through kernel arguments, using the following:
```default
intel_iommu=on iommu=pt
```
### Huge pages
This feature requires 1GB huge pages to be preconfigured. For example, the
following kernel arguments may be used:
```default
default_hugepagesz=1G hugepagesz=1G hugepages=64
```
Openstack instances that leverage DPDK must request huge pages through
flavor extra specs:
```default
openstack flavor set m1.large --property hw:mem_page_size=large
```
Make sure that the flavor ram size is a multiple of the huge page size (1GB).
Instances configured to use huge pages will be connected to the DPDK datapath
through `vhost-user` interfaces, as opposed to the standard tap devices.
See the [Openstack DPDK documentation](https://docs.openstack.org/neutron/latest/admin/config-ovs-dpdk.html) for more details.
### Network configuration
Physical network interfaces that leverage DPDK are expected to be connected to
OVS bridges, either directly or through bonds.
Make sure to configure the bridges using Netplan or MAAS before enabling DPDK.
Canonical Openstack will pivot the configuration from the system OVS
installation to the snap based OVS service. This requires a currently
[unreleased Netplan change](https://github.com/canonical/netplan/pull/549).
At the same time, the DPDK devices will be persistently bound to the
configured DPDK-compatible driver (vfio-pci by default), at which point
the interfaces will no longer be visible to the host.
As such, the charm will remove the DPDK interfaces from the Netplan
configuration and move bond definitions to OVS.
```default
$ sudo /snap/bin/ovs-vsctl show
53cdbac9-b6e4-40c5-8c12-71a874f3a606
Bridge br0
fail_mode: standalone
datapath_type: netdev
Port br0
Interface br0
type: internal
Port bond0
Interface dpdk-eth1
type: dpdk
options: {dpdk-devargs="0000:06:00.0"}
Interface dpdk-eth2
type: dpdk
options: {dpdk-devargs="0000:07:00.0"}
```
### Snapd
`snapd` 2.72 is required, providing the necessary snap permissions.
## Manual mode
As mentioned in the previous section, network interfaces used with DPDK must
be connected to OVS bridges using Netplan:
```default
bridges:
br0:
macaddress: "00:16:3e:c0:43:a8"
mtu: 1500
interfaces:
- bond0
parameters:
forward-delay: "15"
stp: false
openvswitch: {}
bonds:
bond0:
macaddress: "00:16:3e:c0:43:a8"
mtu: 1500
interfaces:
- eth1
- eth2
```
DPDK can be configured using the following command:
```default
sunbeam configure dpdk
```
The user will need to specify which network interfaces are going to connected
to the DPDK datapath and the amount of system resources to allocate.
Example:
```default
$ sunbeam configure dpdk
Enable OVS DPDK data path, handling packets in userspace. It provides improved performance compared to
the standard OVS kernel data path. DPDK capable network interfaces are required.
Enable and configure DPDK [y/n] (n): y
Configuring DPDK physical interfaces.
WARNING: the specified interfaces will be reconfigured to use a DPDK-compatible driver (vfio-pci by
default) and will no longer be visible to the host.
Any bonds and bridges defined in MAAS/Netplan will be updated to use the new DPDK OVS port.
DPDK candidate interfaces:
* Intel Corporation Ethernet Controller X550 (eno2)
* Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0)
* Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0d1)
Enable interface DPDK mode? Intel Corporation Ethernet Controller X550 (eno2) [y/n] (n): y
Enable interface DPDK mode? Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0) [y/n] (n): y
Enable interface DPDK mode? Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0d1) [y/n]
(n): y
The specified number of cores will be allocated to OVS datapath processing, taking into account the
NUMA location of physical DPDK ports. Isolated cpu cores must be preconfigured using kernel parameters.
The number of cores allocated to OVS datapath processing (2):
The specified number of cores will be allocated to OVS control plane processing, taking into account
the NUMA location of physical DPDK ports. Isolated cpu cores must be preconfigured using kernel
parameters.
The number of cores allocated to OVS control plane processing (2):
The total amount of memory in MB to allocate from huge pages for OVS DPDK. The memory will be
distributed across NUMA nodes based on the location of the physical DPDK ports. Currently uses 1GB
pages, make sure to specify a multiple of 1024 and preallocate enough 1GB pages.
The amount of memory in MB allocated to OVS from huge pages (2048): 2048
The DPDK-compatible driver used for DPDK physical ports (vfio-pci):
```
## MAAS mode
Each MAAS network interface connected to the DPDK datapath must contain the
neutron:dpdk tag. Also, it should be connected to an OVS bridge defined in
MAAS, either directly or through a bond.
Apart from that, DPDK can be enabled and configured similarly to the
manual (local) mode.
## Manifest configuration
The DPDK settings can be provided through the Canonical Openstack manifest,
for example:
```default
core:
config:
dpdk:
enabled: true
control_plane_cores: 2
dataplane_cores: 2
memory: 2048
driver: vfio-pci
ports:
my-node.maas:
- eno3
- eno4
```
## Openstack instances using DPDK
Openstack instances must be configured to use huge pages in order to leverage
DPDK.
```default
openstack flavor set m1.large --property hw:mem_page_size=large
```
The instances will then be connected to the DPDK datapath using `vhost-user`
ports:
```default
$ sudo openstack-hypervisor.virsh dumpxml instance-0000000d | grep -i vhost -A 7
$ sudo openstack-hypervisor.ovs-vsctl show
Bridge br-int
fail_mode: secure
datapath_type: netdev
Port vhu90ab19fb-57
Interface vhu90ab19fb-57
type: dpdkvhostuserclient
options: {vhost-server-path="/var/snap/openstack-hypervisor/common/run/libvirt/vhu90ab19fb-57"}
```
## Disabling DPDK
The DPDK feature may be disabled using the following command. Simply specify
“n” when prompted in order to disable DPDK.
```default
sunbeam configure dpdk
Enable OVS DPDK data path, handling packets in userspace. It provides improved performance compared to
the standard OVS kernel data path. DPDK capable network interfaces are required.
Enable and configure DPDK [y/n] (y): n
```
By doing so, the OVS bridges will be set to use the standard system datapath
instead of `netdev` (DPDK).
Note that as part of the DPDK enablement, physical port configuration is moved
from Netplan to OVS and the interfaces are persistently bound to the DPDK
compatible driver (`vfio-pci` by default) using `driverctl`. Those steps
are not reverted automatically, the user may have to manually redefine
bonds and remove the driver overrides. Unbinding the `vfio-pci` driver may
require a host reboot.
At the same time, existing instances will continue to use `vhost-user`
interfaces. Either rebuild or migrate those instances to reconfigure the
port attachments.
# index.html.md
# Using an existing Juju controller
Canonical OpenStack can use an existing Juju controller during bootstrap
instead of deploying a Juju controller within the Canonical OpenStack
deployment.
This allows operators to make use of an existing Juju controller that
could be used to control many Juju deployments of different types of
services - including multiple Canonical OpenStack deployments!
## Register the Juju controller
[Register an existing Juju controller](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-external-juju-controllers.md)
in Sunbeam.
#### NOTE
Ensure a dedicated user is created in the external Juju controller and has
`superuser` permissions granted on this controller.
## Bootstrap with registered Juju controller
Use the option `--controller` with the bootstrap command to make use
of the previously registered Juju controller.
In local mode the roles for the machine still need to be provided during
bootstrap:
```default
sunbeam cluster bootstrap --role compute,storage,control \
--accept-defaults --controller prod-controller-01
```
In MAAS mode the roles are determined by tags on the machines being
deployed, so the roles option is not used:
```default
sunbeam cluster bootstrap --controller prod-controller-01
```
## Example external Juju configuration
An external Juju controller can be bootstrapped on top of a LXD cluster, for example, running
across the same machines that are used in the Canonical OpenStack deployment.
Please refer to the [Bootstrap highly available Juju controller on top of a LXD cluster](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/bootstrap-highly-available-juju-controller-on-top-of-a-lxd-cluster.md) section of this documentation for a detailed procedure on how to accomplish this goal.
# index.html.md
# Managing deployment manifests
This page shows how to manage deployment manifests. For an overview of
manifests, see the [Deployment manifest](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/deployment-manifest.md) page.
#### NOTE
Looking to use a manifest from an edge deployment? Take a look at
[Manifest for non-stable deployments](#manifest-for-non-stable-deployments-4).
## List manifests
To list all manifests, run the following command:
```default
sunbeam manifest list
```
Sample output:
| ID | Applied Date |
|----------------------------------|---------------------|
| c6a6d2ab47ac4c21308483e567d64b04 | 2024-02-05 12:17:59 |
| e446b42859f461e690d66b4d233c1ear | 2024-02-06 07:39:38 |
## Show a manifest
To view the content of a manifest, run the following command:
```default
sunbeam manifest show
```
Sample output:
```text
software:
charms:
keystone-k8s:
channel: 2024.1/candidate
glance-k8s:
channel: 2024.1/candidate
```
To get the latest manifest, use the keyword `latest` instead of the
manifest ID:
```default
sunbeam manifest show latest
```
## Generate a manifest
A manifest file can be generated using the below command:
```default
sunbeam manifest generate --manifest-file
```
The generated manifest will be written to ``.
## Manifest for non-stable deployments
Manifest files for the `candidate` and `edge` risks can be found in:
```default
/snap/openstack/current/etc/manifests/candidate|edge.yml
```
A manifest with complete channel information is needed to deploy on
candidate or edge channels.
## Specify a manifest
A manifest is specified by means of the `--manifest` option. There are
three supported use cases.
### Cluster bootstrap
To specify a manifest during the cluster bootstrap process:
```default
sunbeam cluster bootstrap [--role ] [--manifest ] [--accept-defaults]
```
### Cluster refresh
To specify a manifest during a cluster refresh (update) process:
```default
sunbeam cluster refresh [--manifest ] [--clear-manifest] [--upgrade-release]
```
Only components managed via Terraform can be changed (bootstrap options
will be immutable at this point).
#### NOTE
A manifest update must be accompanied by a complete manifest file
(i.e. not a delta).
### Feature enablement
To specify a manifest during the enablement (or post-enablement) of a
feature:
```default
sunbeam enable [--manifest ] []
```
A post-enablement invocation implies a manifest update.
#### NOTE
A manifest update must be accompanied by a complete manifest file
(i.e. not a delta).
# index.html.md
# Register the Juju controller
As a prerequisite, perform the following steps in existing Juju
controller
- [Add a Juju
user](https://juju.is/docs/juju/manage-users).
- [Grant necessary
permissions](https://juju.is/docs/juju/juju-grant) to the Juju
user.
For example:
```text
juju add-user sunbeam
juju grant -c CONTROLLER sunbeam superuser
```
`CONTROLLER` is a name of the Juju controller.
Adding the Juju user will generate a registration token which is
required to register the Juju controller in the Sunbeam deployment.
To register the controller in Sunbeam use the `register-controller`
command:
```default
sunbeam juju register-controller NAME TOKEN
```
`NAME` is an arbitrary name to refer the Juju controller in
Sunbeam.
`TOKEN` is the registration token generated during Juju user creation.
For example, to register an existing controller with the name
`prod-controller-01` using a token generated as detailed above:
```default
sunbeam juju register-controller prod-controller-01 \
Tm90IGEgcGFzc3dkIGlmIHlvdSBjYXJlIHRvIGRlY29kZSB0aGlzCg==
```
# Unregister the Juju controller
To unregister the controller in Sunbeam use the
`unregister-controller` command:
```default
sunbeam juju unregister-controller NAME
```
For example, to unregister an existing controller with the name
`prod-controller-01`:
```default
sunbeam juju unregister-controller prod-controller-01
```
# index.html.md
# Configuring GPU Passthrough
## Overview
The GPU passthrough allows full access and direct control of physical GPU
device in guests. The guests should have the corresponding drivers to use
the devices.
```
Verify the PCI device in the guest:
```default
$ ssh -i /home/ubuntu/snap/openstack/x1/sunbeam ubuntu@172.16.2.115 sudo lspci -nn
...
...
04:00.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] [10de:1db4] (rev a1)
...
...
```
Above example shows the passthrough device Nvidia 3D controller in the guest.
# index.html.md
# Bootstrap highly available Juju controller on top of a LXD cluster
Canonical Juju does not yet support controller HA modeling capabilities when deployed on top
Kubernetes. This means that Canonical OpenStack clouds deployed using the
[manual bare metal provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-the-manual-bare-metal-provider.md)
do not provide HA for all types of governance functions by default. To bypass this
limitation Canonical recommends using an external highly available Juju controller. Such a
controller can be bootstrapped on top of a LXD cluster, for example, running across the same
machines that are used in the Canonical OpenStack deployment.
This how-to guide provides all necessary information on how to perform aforementioned actions.
## Requirements
You will need:
* at least three dedicated physical machines with:
* hardware specifications matching minimum hardware specifications for the *Cloud* node as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* fresh Ubuntu Server 24.04 LTS installed
## Prepare machines
All machines have to be configured first to use [bridges](https://ubuntu.com/server/docs/configuring-networks#bridging-multiple-interfaces) instead of physical network interfaces on the Generic network.
For example, to prepare the *cloud-1* machine from the example configuration section, execute the following commands:
```text
sudo bash -c 'cat < /etc/netplan/config.yaml
network:
bridges:
br0:
addresses:
- 172.16.1.101/24
interfaces:
- eno1
routes:
- to: default
via: 172.16.1.1
nameservers:
addresses:
- 8.8.8.8
search:
- example.com
ethernets:
eno1:
set-name: eno1
eno2:
set-name: eno2
version: 2
EOF'
sudo netplan apply
```
## Set up a LXD cluster
In the first step, set up a [LXD cluster](https://canonical.com/lxd) across at least three machines.
### Bootstrap the cluster
To bootstrap the cluster, execute the `lxd init` command on the first machine in the cluster (aka primary node):
```text
lxd init
```
When prompted, answer some interactive questions. Below is a sample output from the *cloud-1* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md):
```text
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=172.16.1.101]: 172.16.1.101
Are you joining an existing cluster? (yes/no) [default=no]: no
What member name should be used to identify this server in the cluster? [default=cloud-1]: cloud-1
Do you want to configure a new local storage pool? (yes/no) [default=yes]: yes
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]: zfs
Create a new ZFS pool? (yes/no) [default=yes]: yes
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: no
Size in GiB of the new loop device (1GiB minimum) [default=30GiB]: 30GiB
Do you want to configure a new remote storage pool? (yes/no) [default=no]: no
Would you like to connect to a MAAS server? (yes/no) [default=no]: no
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: yes
Name of the existing bridge or host interface: br0
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: yes
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: no
```
Refer to the [LXD documentation](https://documentation.ubuntu.com/lxd/en/latest/) for detailed description of each of those questions and some examples.
### Create registration tokens
Registration tokens have to be created first for the other machine to be able to join the newly bootstrapped cluster.
In order to create a registration token for the new machine, execute the `lxc cluster add` command on the primary node:
```text
lxc cluster add NAME
```
`NAME` is the name of the machine being added.
For example, to create a registration token for the *cloud-2* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section, execute the following command on the *cloud-1* machine:
```text
lxc cluster add cloud-2
```
Sample output (token):
```text
Member cloud-2 join token:
eyJzZXJ2ZXJfbmFtZSI6ImNsb3VkLTIuZXhhbXBsZS5jb20iLCJmaW5nZXJwcmludCI6IjFhZmYyZGQ3ZDhmZmUwZWE1MzliODA2ZWExNmE4NTRlYTBmYmNjZDU1MTJjYjlmMTk1YmU4YTY4ZTZkYzRkNzYiLCJhZGRyZXNzZXMiOlsiY2xvdWQtMS5leGFtcGxlLmNvbTo4NDQzIl0sInNlY3JldCI6ImYxZmIzMzcxOTlmZmRlNmIzMjYwYjQ1NGY5MTBmNTJhMzE3NGE2OTQ2MTAwMzU1OGU2ZmM3YjEyNDA2NmU2ZWIiLCJleHBpcmVzX2F0IjoiMjAyNC0xMS0wNFQxNToxNDoxOC4zMDE4NTEwNThaIn0=
```
Remember the value of the token. It will be needed in the next step of this how-to guide.
### Add machines to the cluster
Now that the cluster has been bootstrapped and registration tokens have been created, other machines should be able to join the cluster.
To join the cluster, execute the `sudo lxd init` command on all remaining machines:
```text
sudo lxd init
```
When prompted, answer some interactive questions. Below is a sample output from the *cloud-2* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md):
```text
Installing LXD snap, please be patient.
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=172.16.1.102]: 172.16.1.102
Are you joining an existing cluster? (yes/no) [default=no]: yes
Do you have a join token? (yes/no/[token]) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6ImNsb3VkLTIiLCJmaW5nZXJwcmludCI6IjI5Y2UzNzJmYzVkZDg4ODE3NmMxNTNmYTc2OGJlOGJhMjIyNWQ1MGY5NWY2NmUwZTdlNDc4YzM3ODA1Y2U5MmIiLCJhZGRyZXNzZXMiOlsiMTcyLjE2LjEuMTAxOjg0NDMiXSwic2VjcmV0IjoiNjAxNjZmMDY0ODg4Y2ZkY2U1NzZiODgzMmYwYjRlNmVhYzZiOWY1MTU4Nzk3ZDE4MWM3YWFmMTAwZTVjY2ZjYSIsImV4cGlyZXNfYXQiOiIyMDI0LTExLTA0VDE1OjQ4OjU1LjQxMjg1NTg4OFoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local":
Choose "source" property for storage pool "local":
Choose "zfs.pool_name" property for storage pool "local":
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: no
```
Refer to the [LXD documentation](https://documentation.ubuntu.com/lxd/en/latest/) for detailed description of each of those questions and some examples.
### Verify cluster setup
To verify cluster setup, execute the `lxc cluster list` command on any machine in the cluster:
```text
lxc cluster list
```
You should be able to see all machines being used.
Sample output (based on the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section):
```text
+---------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+---------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cloud-1 | https://172.16.1.101:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+---------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cloud-2 | https://172.16.1.102:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+---------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cloud-3 | https://172.16.1.103:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+---------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
```
### Set trust password
Finally, set a trust password so that the cluster can later be registered as a Juju cloud by executing the following command on the primary node:
```text
lxc config set core.trust_password PASSWORD
```
`PASSWORD` is the trust password.
For example:
```text
lxc config set core.trust_password mytrustpassword
```
## Bootstrap Juju controllers
In the next step, bootstrap highly available [Juju controllers](https://juju.is/) across all machines in the cluster.
### Create system account
#### NOTE
Canonical OpenStack cannot be installed under the same system account that is used to perform the initial bootstrap of the external Juju controller. As a result, dedicated system account has to be created first.
To create a dedicated system account and to switch into it, execute the following commands on the primary node:
```text
sudo groupadd bootstrap
sudo useradd -m -g bootstrap -s /bin/bash bootstrap
sudo usermod -a -G lxd,sudo bootstrap
sudo passwd bootstrap
sudo -i
su bootstrap
cd
```
### Install the snap
Then, install the `juju` snap:
```text
sudo snap install juju
```
### Register the LXD cluster as a Juju cloud
Later, register the newly bootstrapped LXD cluster as a Juju cloud by performing the following actions.
Add the LXD cluster to the local LXC config:
```text
lxc remote add NAME IP --password PASSWORD
```
`NAME` is the name of the LXD cluster.
`IP` is the IP address of the primary node in the cluster.
`PASSWORD` is the trust password that was set in one of the previous steps.
When prompted, type `y`.
For example, to register the LXD cluster from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section as `mylxdcluster` cloud, execute the following commands:
```text
$ lxc remote add mylxdcluster 172.16.1.101 --password mytrustpassword
Certificate fingerprint: 29ce372fc5dd888176c153fa768be8ba2225d50f95f66e0e7e478c37805ce92b
ok (y/n/[fingerprint])? y
```
You should now be able to see `mylxdcluster` on the list of available Juju clouds:
```text
$ juju clouds
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.
You can bootstrap a new controller using one of these clouds...
Clouds available on the client:
Cloud Regions Default Type Credentials Source Description
localhost 1 localhost lxd 0 built-in LXD Container Hypervisor
mylxdcluster 1 default lxd 0 built-in LXD Cluster
```
### Bootstrap a Juju controller
To bootstrap a Juju controller on the `mylxdclluster` cloud, execute the following command on the primary node:
```text
juju bootstrap mylxdcluster
```
One finished, you should be able to see the following message on the screen:
```text
Bootstrap complete, controller "mylxdcluster-default" is now available
Controller machines are in the "controller" model
Now you can run
juju add-model
to create a new model to deploy workloads.
```
### Make the controller highly available
To make the controller highly available, execute the following command on the primary node:
```text
juju enable-ha
```
Sample output:
```text
maintaining machines: 0
adding machines: 1, 2
```
The rest now happens in the background. Once finished, you should be able to see your Juju controller being highly available (indicated by `3` under the `HA` column):
```text
$ juju controllers --refresh
Controller Model User Access Cloud/Region Models Nodes HA Version
mylxdcluster-default* - admin superuser mylxdcluster/default 1 3 3 3.5.4
```
#### WARNING
**Bug 1969667**
At the moment, due to [lp1969667](https://bugs.launchpad.net/juju/+bug/1969667), LXC containers hosting Juju controller units do not get distributed equally across all nodes in the LXD cluster by default.
To workaround the aforementioned issue, run the `lxc list` command first:
```text
lxc list
```
Sample output:
```text
+---------------+---------+---------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-0 | RUNNING | 172.16.1.248 (eth0) | | CONTAINER | 0 | cloud-1 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-1 | RUNNING | 172.16.1.249 (eth0) | | CONTAINER | 0 | cloud-2 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-2 | RUNNING | 172.16.1.250 (eth0) | | CONTAINER | 0 | cloud-2 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
```
As you can see the `juju-e4ce90-2` container runs on the `cloud-2` node, while it should run on the `cloud-3` node instead.
To move the `juju-e4ce90-2` container from `cloud-2` to `cloud-3`, execute the following commands:
```text
lxc stop juju-e4ce90-2
lxc move juju-e4ce90-2 --target cloud-3
lxc start juju-e4ce90-2
```
At this point you should be able to see all three containers being equally distributed across all the nodes forming the LXD cluster:
```text
$ lxc list
+---------------+---------+---------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-0 | RUNNING | 172.16.1.248 (eth0) | | CONTAINER | 0 | cloud-1 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-1 | RUNNING | 172.16.1.249 (eth0) | | CONTAINER | 0 | cloud-2 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
| juju-e4ce90-2 | RUNNING | 172.16.1.250 (eth0) | | CONTAINER | 0 | cloud-3 |
+---------------+---------+---------------------+------+-----------+-----------+----------+
```
### Create necessary credentials for the Sunbeam client
To be able to use the newly bootstrapped, highly available Juju controller in the Sunbeam client, [add a new user](https://juju.is/docs/juju/manage-users#add-a-user) to the controller and [grant necessary permissions](https://juju.is/docs/juju/juju-grant) (`superuser`) to this user on the controller.
To add a new user, run:
```text
juju add-user sunbeam
```
Sample output:
```text
User "sunbeam" added
Please send this command to sunbeam:
juju register MHwTB3N1bmJlYW0wPBMSMTcyLjE2LjEuMTIxOjE3MDcwExIxNzIuMTYuMS4xMjI6MTcwNzATEjE3Mi4xNi4xLjEyMzoxNzA3MAQgJIknLboGwWOWObzGW1NFQ45z_TnBIEKt5kwfDL7ZSLsTD215Y2xvdWQtZGVmYXVsdBMA
"sunbeam" has not been granted access to any models. You can use "juju grant" to grant access.
```
Remember the value of the token from the output as it will be needed in next steps.
To grant the user necessary permissions, run:
```text
juju grant -c mylxdcluster-default sunbeam superuser
```
## Register Juju controller in the Sunbeam client
First, log out from the `bootstrap` account:
```text
exit
```
To register `mylxdcluster-default` controller in the Sunbeam client, execute the following command:
```text
sunbeam juju register-controller mylxdcluster-default TOKEN
```
Replace `TOKEN` with the token obtained when creating the `sunbeam` user.
For example:
```text
sunbeam juju register-controller mylxdcluster-default MHwTB3N1bmJlYW0wPBMSMTcyLjE2LjEuMTIxOjE3MDcwExIxNzIuMTYuMS4xMjI6MTcwNzATEjE3Mi4xNi4xLjEyMzoxNzA3MAQgJIknLboGwWOWObzGW1NFQ45z_TnBIEKt5kwfDL7ZSLsTD215Y2xvdWQtZGVmYXVsdBMA
```
At this point, you can bootstrap Canonical OpenStack cluster with Sunbeam while using the
`mylxdcluster-default` controller.
For example:
```text
sunbeam cluster bootstrap --role control,compute,storage --controller mylxdcluster-default
```
# index.html.md
# Configuring vTPM
## Overview
Virtual Trusted Platform Module (vTPM) support allows OpenStack instances to
use an emulated TPM device.
## Prerequisites
* Enable the [Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md) feature.
* Enable the [Secrets as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md) feature.
* Use an image that supports UEFI firmware.
## Operations
Configure a flavor and an image for vTPM usage.
### Validate compute host support
List the TPM traits advertised by a compute host:
```default
COMPUTE_UUID=$(openstack resource provider list --name -f value -c uuid)
openstack resource provider trait list $COMPUTE_UUID | grep SECURITY_TPM
```
The command should return the supported TPM versions:
```default
| COMPUTE_SECURITY_TPM_1_2 |
| COMPUTE_SECURITY_TPM_2_0 |
```
### Flavor properties
Configure a flavor with the TPM version and model to expose to the instance:
To set the properties on an existing flavor, run:
```default
openstack flavor set FLAVORNAME \
--property hw:tpm_version=2.0 \
--property hw:tpm_model=tpm-crb
```
To create a new flavor with the same properties, run:
```default
openstack flavor create FLAVORNAME \
--ram RAM \
--disk DISK \
--vcpus VCPUS \
--property hw:tpm_version=2.0 \
--property hw:tpm_model=tpm-crb
```
#### NOTE
The `tpm-crb` model is only compatible with TPM version `2.0`.
### Image properties
Configure the image to use UEFI firmware:
```default
openstack image set --property hw_firmware_type=uefi IMAGENAME
```
### Launch an instance
Launch an instance using the flavor and image configured above:
```default
openstack server create \
--flavor FLAVORNAME \
--image IMAGENAME \
--network NETWORK \
SERVERNAME
```
### Verify vTPM in the guest
After the instance boots, log in to the guest and check for TPM devices:
```default
ls -l /dev/tpm*
```
The guest should show devices such as `/dev/tpm0` and `/dev/tpmrm0`.
## Limitations
* The image must use UEFI firmware.
* The `tpm-crb` model requires TPM version `2.0`.
## References
For more information about vTPM in OpenStack, see the upstream Nova
documentation:
* [Emulated Trusted Platform Module](https://docs.openstack.org/nova/latest/admin/emulated-tpm.html)
* [Extra Specs](https://docs.openstack.org/nova/latest/configuration/extra-specs.html)
* [Useful image properties](https://docs.openstack.org/glance/latest/admin/useful-image-properties.html)
# index.html.md
# Configuring SR-IOV
## Overview
The Single Root I/O Virtualization (SR-IOV) specification allows splitting
a single physical port (called physical function or PF) into multiple virtual
ports known as virtual functions or VFs.
VFs as well as PFs can be assigned to Openstack instances, bypassing the
host network stack and significantly improving performance while reducing
host resource usage.
Hardware offloading, also known as switchdev mode, is a mechanism that
enables Open vSwitch (OVS) flows to be offloaded directly to the Virtual
Function (VF). This approach combines the performance advantages of SR-IOV
with the flexibility of OVS, allowing VFs to be added to the OVS bridge. As
a result, it supports the full OVN stack and facilitates overlay tenant
networks that rely on tunneling protocols such as VXLAN or Geneve.
sriov:no-physnet
```
Use Curtin scripts to prepare the prerequisite SR-IOV configuration as described
in the [previous section](#sriov-prerequisites). Also ensure that MAAS is configured
to apply the necessary kernel parameters.
Similarly to manual mode, the SR-IOV configuration can be modified using the
following command:
```default
sunbeam configure sriov
```
```
If hardware offloading is available, the device will be added to the `br-int`
bridge:
```default
sudo openstack-hypervisor.ovs-vsctl show
# output #
f9b527db-207c-453d-bcda-482610541462
Bridge br-ex
datapath_type: system
Port br-ex
Interface br-ex
type: internal
Port patch-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055-to-br-int
Interface patch-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055-to-br-int
type: patch
options: {peer=patch-br-int-to-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055}
Bridge br-int
fail_mode: secure
datapath_type: system
Port enp2s0f0r3
Interface enp2s0f0r3
Port tap578cb555-00
Interface tap578cb555-00
Port br-int
Interface br-int
type: internal
Port patch-br-int-to-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055
Interface patch-br-int-to-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055
type: patch
options: {peer=patch-provnet-4cb61b1f-86a8-4c50-956a-04d8358ce055-to-br-int}
Port tapdcf0ee2d-f8
Interface tapdcf0ee2d-f8
ovs_version: "3.5.0"
```
## Disabling SR-IOV
The same command may also be used to disable the SR-IOV functionality.
Specify “n” for each interface that should no longer be used with SR-IOV.
```default
sunbeam configure sriov
Found the following SR-IOV capable devices:
[ ] Intel Corporation Ethernet Controller X550 (eno1) [physnet: None]
[X] Intel Corporation Ethernet Controller X550 (eno2) [physnet: physnet1]
[ ] Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0) [physnet: None]
Add network adapter to PCI whitelist? Intel Corporation Ethernet Controller X550 (eno1) [y/n] (n): n
Add network adapter to PCI whitelist? Intel Corporation Ethernet Controller X550 (eno2) [y/n] (y): n
Add network adapter to PCI whitelist? Mellanox Technologies MT27520 Family [ConnectX-3 Pro] (enp94s0) [y/n] (n): n
```
Existing instances will not be modified, consider removing VF attachments
manually to avoid subsequent port binding failures.
# index.html.md
# Managing OpenStack Identity Providers
## Overview
Perhaps one of the most important user facing features of any cloud is the authentication, authorization
and service discovery workflows that allow users to log into their account and create resources.
OpenStack, through the use of Keystone provides a plugable architecture for authentication while still
retaining the responsibility for authorization and service discovery.
In this document we’ll walk through enabling additional authentication back-ends for keystone by integrating
with OpenID Connect providers such as Google, Okta, Entra ID or [Canonical Identity Platform](https://charmhub.io/topics/canonical-identity-platform).
## Implementation
Keystone does not directly support the needed authentication workflows needed for OpenID or SAML2. Rather,
it delegates that responsibility to Apache2, which then passes onto keystone the result of the authentication
via a set of variables which hold details about the user (full name, email, remote user ID, etc).
Keystone then uses this information to match the authenticated user to a project, based on a set of
rules configured by the cloud operator (detailed later).
Configuring federation has two major components:
* Making an IdP available to Keystone via Apache2 configurations.
* Enabling the IdP in OpenStack by creating the required keystone resources using the openstack command
Canonical OpenStack is responsible for the first part of that configuration process, which involves adding the needed URLs
in Apache2 and secrets to enable the authentication workflows. Making use of the newly enabled
capabilities falls on the cloud administrator, by leveraging standard openstack commands.
## Requirements
Before we begin, make sure that you have enabled TLS through the use of Vault. Virtually all providers require that
the redirect URL is TLS enabled. Moreover, some providers require that the redirect URL uses a fully qualified domain
name and not an IP address. This means that for the external host name you must also set a valid hostname. Both of
these operations can be done by enabling Vault.
## Adding an Identity provider
There are two types of relations supported by Canonical OpenStack:
* External providers (Google, Okta, Entra ID) for both OpenID connect and SAML2
* [Canonical Identity Platform](https://charmhub.io/topics/canonical-identity-platform) which is expected to be deployed in a different juju model.
### Enabling Canonical Identity Platform as an IdP
For the purpose of this document we will assume you have a model called iam which contains a deployment of Canonical Identity Platform.
Integration is done by consuming an offer for the oauth endpoint of the [Hydra](https://charmhub.io/hydra) charm. If your
identity platform uses a custom CA, you also need to consume the send-ca-cert endpoint deployed in your iam model.
Create the offers:
```default
juju offer iam.hydra:oauth
juju offer iam.self-signed-certificates:send-ca-cert
```
Get the offer URLs. We’ll need them when creating the config file:
```default
HYDRA_OFFER=$(juju list-offers --format=json -m iam hydra | jq -r '.hydra.["offer-url"]')
SEND_CA_CERT_OFFER=$(
juju list-offers \
--format=json \
-m iam \
self-signed-certificates | jq -r '.["self-signed-certificates"].["offer-url"]')
```
Create the config file:
```default
cat << EOF > canonical-iam.yaml
oauth_offer: $HYDRA_OFFER
cert_offer: $SEND_CA_CERT_OFFER
EOF
```
And finally add the provider:
```default
sunbeam identity provider add \
canonical openid canonical-identity-platform \
--config canonical-iam.yaml
```
In the above example we added a new identity provider of type canonical with the openid protocol and the name canonical-identity-platform.
the name is important in the case of providers of type canonical as the name is also used as a label in Horizon when selecting the IdP. So make
sure to use a descriptive name.
Now we can list the providers:
```default
sunbeam identity provider list
+-----------------------------+------------+----------+------------+
| Name | Provider | Protocol | Remote ID |
+-----------------------------+------------+----------+------------+
| Keystone Credentials | Built-in | keystone | N/A |
| canonical-identity-platform | canonical | openid | N/A |
+-----------------------------+------------+----------+------------+
```
At this point the configurations exist in apache, keystone and horizon that enable a cloud administrator to configure it as an authentication backend. We will
cover that part later.
### External IdPs
Canonical OpenStack currently supports the following IdP types:
* [Google (OIDC)](https://developers.google.com/identity/openid-connect/openid-connect)
* [Google (SAML2)](https://cloud.google.com/chronicle/docs/soar/admin-tasks/saml-soar-only/saml-configuration-for-g-suite)
* [Okta (OIDC)](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm)
* [Okta (SAML2)](https://developer.okta.com/docs/guides/add-an-external-idp/saml2/main/)
* [Entra ID (OIDC)](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc#enable-id-tokens)
* [Entra ID (SAML2)](https://learn.microsoft.com/en-us/entra/architecture/auth-saml)
* Generic (SAML2 and OIDC)
Each of these provider types require specific configurations in order to enable them and each have their own procedure for creating the client credentials needed to initiate the authentication workflow.
You will have to consult the official documentation for each, in order to generate the required client credentials. The configuration formats that Canonical OpenStack requires are detailed below. The configuration
files are in yaml format.
When creating an OpenID Connect integration meant to be used with Canonical OpenStack, you will need the redirect URL that the IdP needs to call back into when a user authenticates. To display the redirect URL, you can run
the following command:
```default
sunbeam identity provider get-oidc-redirect-url
https://sunbeam.example.com/openstack-keystone/v3/OS-FEDERATION/protocols/openid/redirect_uri
```
For SAML2 you will need to know the metadata URL of the Service Provider (Keystone in our case). The metadata URL will return the SP XML for keystone, where you can find
the signing certificate that keystone will use, the single sign out URL and the Assertion Consumer Service URL. You will need this information to set up the SAML2 application
in your provider of choice.
The metadata URL for a particular provider can be inferred from the FQDN of keystone, the provider name and the provider protocol.
If we have a provider named entra that uses saml2 and our FQDN is sunbeam.example.com then the metadata URL will be:
```default
https://sunbeam.example.com/openstack-keystone/v3/OS-FEDERATION/identity_providers/entra/protocols/saml2/auth/mellon/metadata
```
Note, the schema **must** be **https** and you **should** have a fully qualified domain name configured instead of an IP address. Depending on IdP, this might be a requirement (Google for example). If that is not the case,
you should enable TLS in sunbeam, using Vault.
#### SAML2 special consideration
When creating a SAML2 entry in Canonical OpenStack, there is a bit of a chicken and egg situation. The application needs to exist in the provider of choice before you
can add it to Canonical OpenStack, but you also need the information in the metadata XML we offer to configure the application in the IDP of choice. Luckily, the information
you use when creating the application does not need to be accurate. You will be able to create the application even with placeholder values. Once you create the application, you
can add it to Canonical OpenStack. Once added, you will be able to get the values from the metadata URL mentioned above and edit the application in the IDP of choice.
Another important consideration is that for SAML2 you will need to make sure you’ve added an x509 signing certificate and the corresponding key:
```default
sunbeam identity set-saml-x509 /path/to/cert.pem /path/to/key.pem
```
#### Google config format (OIDC)
There are two mandatory configuration parameters and one optional parameter:
* client-id - mandatory
* client-secret - mandatory
* label - optional
Example config:
```default
client-id: client_id_obtained_from_your_console
client-secret: client_secret_associated_with_the_id
label: "Log in with Google (OIDC)"
```
#### Google config format (SAML2)
There is one mandatory configuration parameter and one optional parameter:
* app-id - mandatory
* label - optional
Example config:
```default
app-id: saml2_app_id
label: "Log in with Google (SAML2)"
```
#### Okta config format (OIDC)
There are three mandatory configuration parameters and one optional parameter:
* client-id - mandatory
* client-secret - mandatory
* okta-org - mandatory
* label - optional
Example config:
```default
client-id: client_id_obtained_from_your_console
client-secret: client_secret_associated_with_the_id
okta-org: dev-123456
label: "Log in with Okta"
```
#### Okta config format (SAML2)
There are two mandatory configuration parameters and one optional parameter:
* app-id - mandatory
* okta-org - mandatory
* label - optional
Example config:
```default
app-id: app_id_goes_here
okta-org: dev-123456
label: "Log in with Okta (SAML2)"
```
#### Entra ID config format (OIDC)
There are three mandatory configuration parameters and one optional parameter:
* client-id - mandatory
* client-secret - mandatory
* microsoft-tenant - mandatory
* label - optional
Example config:
```default
client-id: client_id_obtained_from_your_console
client-secret: client_secret_associated_with_the_id
microsoft-tenant: tenant-uuid-goes-here
label: "Log in with Entra ID (OIDC)"
```
#### Entra ID config format (SAML2)
There are two mandatory configuration parameters and one optional parameter:
* app-id - mandatory
* microsoft-tenant - mandatory
* label - optional
Example config:
```default
app-id: app_id_goes_here
microsoft-tenant: tenant-uuid-goes-here
label: "Log in with Entra ID (SAML2)"
```
#### Generic (OIDC)
The generic provider allows you to configure any OIDC compatible provider.
There are three mandatory parameters and one optional parameter:
* client-id - mandatory
* client-secret - mandatory
* issuer-url - mandatory
* label - optional
Example config:
```default
client-id: client_id_obtained_from_your_console
client-secret: client_secret_associated_with_the_id
issuer-url: https://oidc.example.com
label: "Log in with My OpenID connect provider"
```
A note about the issuer-url. This URL identifies the provider. It is also the URL from which we get the OpenID connect configuration.
The issuer URL, must have the well-known openid configuration URL available. This URL can be constructed by appending
/.well-known/openid-configuration to the issuer URL.
Example:
```default
https://accounts.google.com/.well-known/openid-configuration
```
In the above example, https://accounts.google.com is the issuer-url.
For the generic OpenID connect there is no option to specify a custom CA certificate chain to validate the issuer-url. You will need to use
a certificate issued by a CA that your deployment already trusts.
#### Generic (SAML2)
Similar to the OIDC generic provider, the generic SAML2 provider allow you to configure any SAML2 compliant IDP, as long as you know the metadata URL.
There is only one mandatory configuration parameter for the SAML2 provider and two optional parameters.
* metadata-url - mandatory
* ca-chain - optional
* label - optional
Example config:
```default
metadata-url: https://saml2.example.com/metadata
ca-chain: base64-encoded-ca-chain-goes-here
label: "Log in with My Custom SAML2 IDP"
```
The metadata URL must contain a XML response that identifies the IDP. The XML must contain the remote entityID, as well as the signing x509 keys of the remote IDP.
The value of the entityID property must be used when defining the IDP in Canonical OpenStack as the remote ID.
### Adding an external IdP
Adding an external IdP is similar to adding a Canonical Identity Platform provider:
```default
sunbeam identity provider add \
google openid my-google-idp \
--config google.yaml
```
Now we can list the providers:
```default
sunbeam identity provider list
+-----------------------------+------------+----------+------------------------------+
| Name | Provider | Protocol | Remote ID |
+-----------------------------+------------+----------+------------------------------|
│ Keystone Credentials │ Built-in │ keystone │ N/A │
| canonical-identity-platform | canonical | openid | N/A |
│ my-google-idp │ google │ openid │ https://accounts.google.com │
+-----------------------------+------------+----------+------------------------------|
```
Adding a SAML2 or OIDC provider has a similar procedure for all above mentioned options.
Make a note of the name of the provider and of the protocol. We will use them in the next steps to enable these providers in keystone.
Note, you should already see them in Horizon, but you will only be able to use them after we’ve mapped them to domains and projects. Examples below.
### Removing a provider
Removing a provider is a matter of running:
```default
sunbeam identity provider remove my-google-idp --yes-i-mean-it
```
Note, this will not remove any resources created by the cloud administrator using the openstack command.
### Making use of the new providers
Now that we’ve made the providers available to the cloud, we can enable them in keystone, map them to a domain and create rules on how users should
be mapped to projects.
You can create a new domain or you can use an existing domain to map it to the IdP. For the purposes of this guide, we’ll create a new one:
```default
openstack domain create \
--description="Federated Google domain" \
google
```
Get the issuer URL for the desired IdP. In this case we’ll go with my-google-idp from the output above:
```default
REMOTE_ID=$(sunbeam identity provider list \
--format=yaml | yq -r '.openid."my-google-idp".remote_id')
```
Note, if you’re configuring a saml2 IDP, you will need to adapt the yq arguments in the above command.
Create the identity provider in Keystone:
```default
openstack identity provider create \
--remote-id $REMOTE_ID \
--domain google \
my-google-idp
```
Note, the name of the identity provider must match the name in the table outputted by sunbeam.
Create a group which we will assign to federated users:
```default
openstack group create federated_users \
--domain google
```
Create a project. The following example creates a project named `federated_project`:
```default
openstack project create \
--domain google \
federated_project
```
Add a role for the group on the project we want to use:
```default
openstack role add \
--group federated_users \
--project federated_project \
--group-domain google \
--project-domain google \
member
```
Next, we need to create some mapping rules between the remote users that come in from the IdP and local openstack users. The rules instruct Keystone how
to automatically create local users and to assign them to groups, projects, domains, etc. You may consult [the official documentation](https://docs.openstack.org/keystone/latest/admin/federation/mapping_combinations.html)
on how to write the rules. In this guide we’ll create a simple rule set which will be used for the openid protocol of the my-google-idp provider to map
users to the group we created above. That will automatically grant them **member** access in the **federated_project** of the **google** domain.
This file can be as complex as you need it to be, based on your needs.
Create a file with the rules:
```default
cat > rules.json <
```
## Post-Deployment Management
### Setting a Theme
To set the theme after bootstrap is complete use the `dashboard theme set` command:
```default
sunbeam dashboard theme set
```
### Clearing a Theme
To clear an existing custom theme and return to the default use the `dashboard theme clear` command:
```default
sunbeam dashboard theme clear
```
### Limitations When Used With a Manifest
When setting/clearing a theme using these imperative commands it will always override you manifest values. However, when a cluster operation is performed that re-evaluates the state of the control plane (f.e. `sunbeam cluster refresh`) the manifest will always take priority over values provided after the fact. As such, when utilizing the manifest to declare your theme these commands should only be used as a “quick test” to ensure that your theme applies correctly before updating the manifest and supplying it on the next cluster operation to ensure that the changes are persistent.
# index.html.md
# Adding AMD SEV enabled Compute Node
Secure Encrypted Virtualization is a technology from AMD enabling encryption of guest memory.
OpenStack provides support to make use of this technology and deploy trusted guests.
Refer [Admin guide](https://docs.openstack.org/nova/latest/admin/sev.html) for more information.
In Canonical OpenStack, AMD SEV enabled compute nodes can be added using the [cluster scale out procedure](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md) by adding compute role to the node.
Canonical OpenStack auto detects the compute node if the node is AMD SEV enabled or not.
For AMD SEV enabled compute nodes, sufficient memory need to be reserved for the host since SEV enabled guests memory pages are pinned in RAM.
To set the reserved memory for the host, update manifest with the following configuration for openstack-hypervisor charm.
```default
core:
config:
software:
charms:
openstack-hypervisor:
config:
reserved-host-memory-mb-for-sev: 8192
```
For manual bare metal provider, pass the updated manifest in join command
```default
cat TOKEN_FILE | sunbeam cluster join --manifest MANIFEST_FILE --role ROLES -
```
For MAAS provider, pass the updated manifest in deploy command
```default
sunbeam cluster deploy --manifest MANIFEST_FILE
```
The configuration will be applied only on AMD SEV enabled compute nodes.
## Operations
Once the cloud is deployed, Operator need to do the following operations
### Flavor properties
Create or set flavors with the property hw:mem_encryption=true.
To create new flavor with the above property, run the command
```default
openstack flavor create FLAVORNAME --ram RAM --disk DISK --vcpus VCPUS --property hw:mem_encryption=true
```
To set property on existing flavor, run the command
```default
openstack flavor set --property hw:mem_encryption=true FLAVORNAME
```
Flavors created by sunbeam configure ending with -sev have already the property added.
### Image properties
Create or set images with the property hw_firmware_type=uefi
To create new image with the above property, run the command
```default
openstack image create --disk-format FORMAT --container-format CFORMAT --file IMAGEFILE --property hw_firmware_type=uefi IMAGENAME
```
To set property on existing image, run the command
```default
openstack image set --property hw_firmware_type=uefi IMAGENAME
```
The [Images Sync Feature](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/images-sync.md) will add the property hw_firmware_type=uefi by default when importing images.
### Launch instance
To launch an SEV encrypted instance, use the flavor and images set with the above properties.
## Limitations
* Live migration is not supported for AMD SEV enabled guests.
# index.html.md
```
For help with command syntax see the documentation for the
[python-openstackclient](https://docs.openstack.org/python-openstackclient/latest/cli/command-list.html)
package.
## Unprivileged vs admin user credentials
Unprivileged user credentials are generated at cloud-configuration time
with the `sunbeam configure` command.
Admin user credentials are generated with the `sunbeam openrc`
command. Here, the file `admin-openrc` is chosen as init file:
```default
sunbeam openrc > admin-openrc
```
# index.html.md
# Multi-region deployments
OpenStack deployments can span across multiple regions while sharing only the
identity (Keystone) and dashboard (Horizon) services.
Commonly used with large deployments, the OpenStack region concept is
quite flexible. These may be geographically distinct locations, providing
regional level fault tolerance.
Alternatively, regions can exist within the same physical site, used to
partition a large cluster and distribute load among critical components like
databases or message brokers.
## Manual bare metal provider
### Region controllers
To accommodate multi-region environments, Canonical OpenStack defines the
region controller role, which is mutually exclusive with all other roles.
Region controllers will only run the shared services, such as Keystone and
Horizon.
When using the manual bare metal provider, a region controller may be
bootstrapped like so:
```default
sunbeam cluster bootstrap --role region_controller
```
For high availability, additional region controllers can be added using
the standard procedure. First, obtain a cluster join token:
```default
sunbeam cluster add $fqdn
```
Then use the token to join the new region controller node:
```default
sunbeam cluster join --role region_controller $token
```
Note that standard Sunbeam deployments can also act as primary regions.
### Secondary regions
The secondary regions are distinct Canonical Openstack deployments, using
separate Kubernetes clusters and Juju controllers.
Shared services such as Keystone and Horizon will be omitted from secondary
regions. These services will be provided by the region controllers and
consumed through cross-controller Juju relations.
In order to access the region controller, each secondary region node will
need a token obtained from the region controllers:
```default
sunbeam cluster add-secondary-region-node $fqdn
```
The token must then be passed to the bootstrap command:
```default
sunbeam cluster bootstrap \
--role control,compute,storage --region-controller-token=$token
```
During bootstrap, make sure to specify a region name other than the one of the
region controller.
A region controller token is also required when joining secondary region nodes.
This means that the join operation requires two tokens: one from the region
controller and one from an existing member of the secondary region.
```default
sunbeam --verbose cluster join --role control,compute,storage \
--region-controller-token $region_ctrl_token $same_region_token
```
## Canonical MAAS provider mode
To deploy region controller nodes, use the `region_controller` machine tag.
Note that a deployment containing region controllers is not allowed
to have other control, compute or storage nodes.
```default
$ sunbeam cluster bootstrap
$ sunbeam cluster deploy
Deployment complete with 0 control, 0 compute and 0 storage nodes.
Region controllers: 1. Total nodes in cluster: 1
```
Secondary regions will reside in separate Sunbeam deployments, using a
region controller token to connect to the primary region.
```default
$ sunbeam cluster bootstrap --region-controller-token=$token
$ sunbeam cluster deploy
```
## Juju cross-controller relations
For the time being, the Juju Terraform provider does not support
[cross-controller relations](https://github.com/juju/terraform-provider-juju/issues/805). As such, these relations must be manually
defined.
```default
controller="sunbeam-controller-region-controller"
# Usually the region controller fqdn
owner="$ownerFqdn"
# Run the following when Sunbeam reaches the following phase:
# ⠼ Deploying OpenStack Control Plane to Kubernetes (this may take a while) ... waiting for services to come online (14/18)
juju switch openstack
juju consume $controller:$owner/openstack.keystone-credentials
juju consume $controller:$owner/openstack.keystone-endpoints
juju consume $controller:$owner/openstack.keystone-ops
juju consume $controller:$owner/openstack.cert-distributor
juju consume $controller:$owner/openstack.horizon-cors-origin
juju integrate keystone-endpoints cinder:identity-service
juju integrate keystone-endpoints glance:identity-service
juju integrate keystone-endpoints neutron:identity-service
juju integrate keystone-endpoints nova:identity-service
juju integrate keystone-endpoints placement:identity-service
juju integrate cert-distributor cinder:receive-ca-cert
juju integrate cert-distributor glance:receive-ca-cert
juju integrate cert-distributor neutron:receive-ca-cert
juju integrate cert-distributor nova:receive-ca-cert
juju integrate cert-distributor placement:receive-ca-cert
juju integrate horizon-cors-origin glance:cors-origin
# Run the following once Sunbeam reaches the following phase:
# ⠸ Deploying OpenStack Hypervisor ...
juju switch admin/openstack-machines
juju consume $controller:$owner/openstack.keystone-credentials
juju consume $controller:$owner/openstack.cert-distributor
juju integrate keystone-credentials openstack-hypervisor:identity-credentials
juju integrate keystone-credentials cinder-volume:identity-credentials
juju integrate cert-distributor openstack-hypervisor:receive-ca-cert
```
# index.html.md
# Operations
* [Cluster upgrades](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#overview)
* [Prerequisites](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#prerequisites)
* [Step 1 - Refresh Kubernetes](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#step-1-refresh-kubernetes)
* [Step 2 - Refresh Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#step-2-refresh-vault)
* [Step 3 - Refresh MySQL](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#step-3-refresh-mysql)
* [Step 4 - Refresh the cluster](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#step-4-refresh-the-cluster)
* [Multi-region deployments](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/cluster-upgrades.md#multi-region-deployments)
* [Deploy a Pure Storage backend](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#overview)
* [Requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#requirements)
* [Inspect the available options](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#inspect-the-available-options)
* [Create the backend configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#create-the-backend-configuration)
* [Deploy the backend](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#deploy-the-backend)
* [Verify the backend](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/deploy-pure-storage-backend.md#verify-the-backend)
* [Enable and deploy a gated storage backend](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md)
* [List the available feature gates](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#list-the-available-feature-gates)
* [Enable the storage backend gate](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#enable-the-storage-backend-gate)
* [Verify that the backend is unlocked](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#verify-that-the-backend-is-unlocked)
* [Review the backend options in the CLI](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#review-the-backend-options-in-the-cli)
* [Deploy the backend](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#deploy-the-backend)
* [Verify the deployment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/enable-a-gated-storage-backend.md#verify-the-deployment)
* [Live Migration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/live-migration.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/live-migration.md#overview)
* [Ensure adequate capacity on the destination host](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/live-migration.md#ensure-adequate-capacity-on-the-destination-host)
* [Live migrate an instance](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/live-migration.md#live-migrate-an-instance)
* [Maintenance mode](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/maintenance-mode.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/maintenance-mode.md#overview)
* [Enabling Maintenance feature](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/maintenance-mode.md#enabling-maintenance-feature)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/maintenance-mode.md#usage)
* [Manage experimental features](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md#overview)
* [List feature gates](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md#list-feature-gates)
* [Enable an experimental feature](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md#enable-an-experimental-feature)
* [Disable an experimental feature](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md#disable-an-experimental-feature)
* [Removing the primary node](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/removing-the-primary-node.md)
* [Remove components from the machine](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/removing-the-primary-node.md#remove-components-from-the-machine)
* [Scaling the cluster in](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-in.md)
* [Scaling the cluster in using the manual bare metal provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-in.md#scaling-the-cluster-in-using-the-manual-bare-metal-provider)
* [Scaling the cluster in using Canonical MAAS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-in.md#scaling-the-cluster-in-using-canonical-maas)
* [Scaling the cluster out](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md)
* [Scaling the cluster out using the manual bare metal provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md#scaling-the-cluster-out-using-the-manual-bare-metal-provider)
* [Scaling the cluster out using Canonical MAAS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md#scaling-the-cluster-out-using-canonical-maas)
* [Backup and Restore](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#overview)
* [s3-integrator](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#s3-integrator)
* [MySQL](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#mysql)
* [Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#vault)
* [K8s control plane backup](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#k8s-control-plane-backup)
* [Juju](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#juju)
* [MAAS deployment access](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#maas-deployment-access)
* [Sunbeam-clusterd](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/backup-and-restore.md#sunbeam-clusterd)
# index.html.md
# Backup and Restore
## Overview
Regular backups of the Sunbeam cluster are a critical component of any robust disaster recovery plan,
ensuring the resilience and continuity of the Canonical OpenStack Cluster deployment. Given that
the procedures described below primarily focus on backing up essential control-plane elements
including application data (MySQL, Vault), the Kubernetes control plane, Juju controller state,
and sunbeam-clusterd.
Unexpected hardware failures, human error, or data corruption can severely compromise the
control plane, leading to extended outages and potential data loss. By maintaining up-to-date
backups, administrators can significantly minimize recovery time objectives (RTO) and restore the
core management services necessary for operating the cloud infrastructure.
## s3-integrator
The Sunbeam cluster, by default, utilizes ceph-rgw within MicroCeph, which provides S3-compatible
object storage capabilities. This built-in functionality can be used to create the S3 buckets
necessary for the backup procedures described here. While this is convenient for initial setup
and testing, it is recommended that for production environments, all critical backups be
stored in an S3-compatible service located outside of the Canonical OpenStack Cluster deployment
itself. Storing backups externally ensures resilience against catastrophic failures that could
affect the entire cloud environment, including the internal Ceph cluster.
For demonstration purposes, the backup procedures outlined in this document will utilize the internal
Ceph Rados Gateway (RGW) provided by the ceph-rgw charm.
```text
juju switch openstack-machines
juju exec -u microceph/leader -- microceph.radosgw-admin user create --uid my-user --display-name my-user
{
"user_id": "my-user",
"display_name": "my-user",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "my-user",
"access_key": "", # save this access key
"secret_key": "", # save this secret key
"active": true,
"create_date": "2026-02-26T20:40:18.959341Z"
}
],
}
# get the endpoint of the ceph-rgw service on openstack model
juju switch openstack
juju run traefik-rgw/leader show-external-endpoints
Running operation 316 with 1 task
- task 317 on unit-traefik-rgw-1
Waiting for task 317...
external-endpoints: '{"traefik-rgw": {"url": "http://"}}'
```
Install a tool like aws-cli\` or s3cmd and configure it with the access key and secret key
obtained from the previous command to interact with the S3 storage provided by ceph-rgw.
```text
sudo snap install aws-cli --classic
aws configure --profile ceph # fill the asked information
aws --profile ceph --endpoint-url http:// s3api create-bucket --bucket mysql
...
# repeat the previous command to create a bucket for each application you want to backup
```
Deploy one s3-integrator application for each application that needs s3-integration. E.g:
```text
juju switch openstack
juju deploy s3-integrator --model openstack mysql-s3-integrator
juju integrate mysql-s3-integrator mysql
...
# deploy and integrate for all necessary apps
```
Run the sync-s3-credentials action to configure the charm
```text
juju run mysql-s3-integrator/leader sync-s3-credentials access-key= secret-key=
...
# do the same for all necessary apps
```
Configure the s3-integrator charm to use the correct bucket for each application
```text
juju config mysql-s3-integrator bucket=mysql s3-uri-style=path endpoint=http:// path=mysql
...
# do the same for all necessary apps
```
## MySQL
### Requirements
* A deployed MySQL K8s cluster
* Access to S3 storage
* Configured settings for S3 storage
* Units in active/idle
* Control-plane units paused to avoid usage of the cluster during **restore** procedure
### Backup
The backup procedure should be executed on secondary MySQL units to avoid impacting the performance
of the primary unit. To get a secondary unit, run the following command:
```text
juju run mysql/leader get-cluster-status
Running operation 196 with 1 task
- task 197 on unit-mysql-2
Waiting for task 197...
status:
clustername: cluster-1e57de179fb5edd8c4e6392a25473b96
clusterrole: primary
defaultreplicaset:
name: default
primary: mysql-2.mysql-endpoints.openstack.svc.cluster.local.:3306
ssl: required
status: ok
statustext: cluster is online and can tolerate up to one failure.
topology:
mysql-0:
address: mysql-0.mysql-endpoints.openstack.svc.cluster.local.:3306
memberrole: secondary
mode: r/o
replicationlagfromimmediatesource: ""
replicationlagfromoriginalsource: ""
role: ha
status: online
version: 8.0.41
mysql-1:
address: mysql-1.mysql-endpoints.openstack.svc.cluster.local.:3306
memberrole: secondary
mode: r/o
replicationlagfromimmediatesource: ""
replicationlagfromoriginalsource: ""
role: ha
status: online
version: 8.0.41
mysql-2:
address: mysql-2.mysql-endpoints.openstack.svc.cluster.local.:3306
memberrole: primary
mode: r/w
role: ha
status: online
version: 8.0.41
topologymode: single-primary
domainname: cluster-set-1e57de179fb5edd8c4e6392a25473b96
groupinformationsourcemember: mysql-2.mysql-endpoints.openstack.svc.cluster.local.:3306
success: "True"
```
It’s possible to see in this case that mysql/0 and mysql/1 are secondary and mysql/2 is primary.
So backups should be run on unit 0 or 1.
```text
juju run mysql/0 create-backup --wait 1m
```
### Restore
To restore it is recommended to stop all control-plane services that might be using the database
before running the restore-backup action. This is to avoid any issues related to data corruption
or inconsistencies during the restore process.
At the moment, there isn’t a charm action to stop all control-plane services at once, so it needs
to be done manually by running on all OpenStack API services:
```bash
# get the container names of all OpenStack API services
kubectl get pods -n openstack -o json | jq -r '
.items[]
| select(
(.metadata.name | test("traefik|rabbitmq|mysql|modeloperator|ovn") | not)
)
| .metadata.name as $pod
| .spec.containers[]
| select(.name != "charm")
| "\($pod) => \(.name)"
'
...
# get the pebble service names for all OpenStack API services
for i in {0..2}; do kubectl -n openstack exec keystone-$i -c keystone -- pebble services; done
# do the same for all necessary apps
# stop the containers of all OpenStack API services
for i in {0..2}; do kubectl -n openstack exec keystone-$i -c keystone -- pebble stop wsgi-keystone; done
# do the same for all necessary apps
```
With all API services stopped, it’s possible to run the restore-backup action on a MySQL unit.
Before that is necessary to scale down the MySQL cluster to 1 replica to ensure data consistency
during the restore process. See the [charmed MySQL documentation](https://canonical-charmed-mysql.readthedocs-hosted.com/8.0/how-to/back-up-and-restore/restore-a-backup/) for more details
```text
juju scale-application mysql 1
```
Then, run the restore-backup action on the unit where you want to restore the backup. E.g:
.. code-block :: text
> juju run mysql/leader restore-backup backup-id=
After restoring all databases, it’s necessary to resume the OpenStack services and scale again
the mysql units.
```text
# start the containers of all OpenStack API services
for i in {0..2}; do kubectl -n openstack exec keystone-$i -c keystone -- pebble start wsgi-keystone; done
# do the same for all necessary apps
juju scale-application mysql 3
```
In case you find mysql-routers on blocked state, it’s necessary to re-launch them by running the following command:
.. code-block :: text
> juju scale-application keystone-mysql-router 0
> juju scale-application keystone-mysql-router 3
After the restoration, MySQL application will be in blocked state with the message:
“Move restored cluster to another S3 repository”. To unblock it, it’s necessary to create a new S3
bucket and configure the mysql-s3-integrator\` charm to use it by running the following command:
.. code-block :: text
> juju config mysql-s3-integrator bucket=
## Vault
### Requirements
* Have a Vault cluster enabled in Sunbeam.
* Units are in active idle state
* Configured settings for S3 storage
* Have saved your unseal keys and root-token in a secure location of your choice
### Backup / Restore
```text
juju run vault/leader create-backup
juju run vault/leader list-backups
juju run vault/leader restore-backup backup-id=
```
## K8s control plane backup
### Requirements
* Have a [velero-operator](https://charmhub.io/velero-operator) deployed
* Have the [infra-backup-operator](https://charmhub.io/infra-backup-operator/docs/tutorial) deployed
* Have access to S3 storage
* Configure s3-integrator
### Backup
```text
juju run velero-operator/0 create-backup \
target=infra-backup-operator:cluster-infra-backup
juju run velero-operator/0 create-backup \
target=infra-backup-operator:namespaced-infra-backup
```
### Restore
```text
# list the backups
juju run velero-operator/0 list-backups
backups:
83503892-a24a-409b-b0df-553dcc2465ec:
app: infra-backup-operator
completion-timestamp: "2025-08-08T20:00:28Z"
endpoint: cluster-infra-backup
model: test-charm-9f0e8dda
name: infra-backup-operator-cluster-infra-backup-pblz2
phase: Completed
start-timestamp: "2025-08-08T20:00:26Z"
85662948-8e5e-4922-8e1c-c5568eafa6e7:
app: infra-backup-operator
completion-timestamp: "2025-08-07T18:42:13Z"
endpoint: cluster-infra-backup
model: test-charm-9f0e8dda
name: infra-backup-operator-cluster-infra-backup-4bm7p
phase: Completed
start-timestamp: "2025-08-07T18:42:10Z"
# restore the backups
juju run velero-operator/0 restore backup-uid=85662948-8e5e-4922-8e1c-c5568eafa6e7
juju run velero-operator/0 restore backup-uid=83503892-a24a-409b-b0df-553dcc2465ec
```
## Juju
### Backup
```text
# export all models
juju export-bundle --model=cos --filename=cos-bundle.yaml
juju export-bundle --model=openstack --filename=openstack-bundle.yaml
...
# backup of controller
juju create-backup --model=${CONTROLLERS_MODEL} --filename=juju-ctrl-backup.tar.gz
# local client configuration
tar -czf juju-credentials.tar.gz ~/.local/share/juju/*
```
### Restore
For restoring there is the [juju-restore](https://github.com/juju/juju-restore/) tool to help.
## MAAS deployment access
See the [Backup and Restore MAAS Deployment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/backup-and-restore-maas-deployment.md) for details.
## Sunbeam-clusterd
### Backup
It’s recommended to create a backup of sunbeam-clusterd data by running the following command:
```text
juju exec -a sunbeam-clusterd -- tar -cvf /home/ubuntu/backup.tar /var/snap/openstack/common/state/database
```
Note that the backup file is created in the home directory of the ubuntu user, so it needs to be
moved to a safe location after the backup is created.
### Restore
If a unit has a corrupted database, it’s possible to restore the backup by running the following command:
```text
# stop the clusterd service before restoring the backup
juju exec -a sunbeam-clusterd -- sudo systemctl stop snap.openstack.clusterd.service
# remove snapshots and segments database files from the corrupted unit
juju exec -u sunbeam-clusterd/{unit} -- rm /var/snap/openstack/common/state/database/snapshot*
juju exec -u sunbeam-clusterd/{unit} -- rm /var/snap/openstack/common/state/database/000000*
# restore the backup on the corrupted unit
juju exec -u sunbeam-clusterd/{unit} -- tar -xvf /home/ubuntu/backup.tar -C /
# start the clusterd service after restoring the backup
juju exec -a sunbeam-clusterd -- sudo systemctl start snap.openstack.clusterd.service
```
# index.html.md
# Live Migration
## Overview
An instance migration is the relocation of an instance from one
hypervisor to another.
When an instance has a live migration performed it is not shut down
during the process. This is useful when there is an imperative to not
interrupt the applications that are running on the instance.
Points to consider:
- network usage may be significantly impacted if block migration mode
is used
- instances with intensive memory workloads may require pausing for
live migration to succeed
## Ensure adequate capacity on the destination host
Oversubscribing the destination host (hypervisor) can lead to service
outages. This is only an issue when a destination host is explicitly
selected by the operator.
The following commands are useful for discovering a instance’s flavor,
listing flavor parameters, and viewing the available capacity of a
destination host:
```text
openstack server show -c flavor
openstack flavor show -c vcpus -c ram -c disk
openstack hypervisor list
openstack host show
```
## Live migrate an instance
Live migration commands require the user to have the admin role.
To live migrate to any hypervisor with sufficient capacity:
```default
openstack server migrate --live-migration
```
To live migrate to a specific hypervisor:
```default
openstack server migrate --live-migration --os-compute-api-version 2.30 --host
```
If the instance has local storage, you must also specify the
`--block-migration` option:
```default
openstack server migrate --block-migration --live-migration
```
# index.html.md
# Scaling the cluster out
Canonical OpenStack scales out, meaning that you can add more machines to the cluster if you need more resources or if you’re designing the cloud for high availability.
Make sure you get familiar with the following sections before proceeding with any instructions listed below:
* [Architecture](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/architecture.md)
* [Design considerations](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/design-considerations.md)
* [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md)
* [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md)
## Scaling the cluster out using the manual bare metal provider
The following section provides instructions on scaling the cluster out with the manual bare metal provider.
### Requirements
You will need:
* Canonical OpenStack already installed using the [manual bare metal provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-the-manual-bare-metal-provider.md)
* one dedicated physical machine with:
* hardware specifications matching minimum hardware specifications as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* fresh Ubuntu Server 24.04 LTS installed
If you can’t provide an unlimited access to the Internet, see the [Manage a proxied environment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-a-proxied-environment.md) section.
### Create a registration token
#### WARNING
Clustering does not support base hostnames. Nodes are only recognized by their **FQDNs**.
A registration token has to be created first for the other machine to be able to join the existing Canonical OpenStack cluster.
In order to create a registration token for the new machine, execute the `sunbeam cluster add` command on the first machine in the cluster (aka primary node):
```text
sunbeam cluster add FQDN --output FILE
```
`FQDN` is a fully qualified domain name (FQDN) of the machine being added.
`FILE` is a name of the file where to save the registration token.
For example, to create a registration token for the *cloud-2* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section, execute the following command on the *cloud-1* machine:
```text
sunbeam cluster add cloud-2.example.com --output cloud-2.asc
```
Sample output:
```text
Token written to file: /home/ubuntu/cloud-2.asc
```
Copy the file with the token (here `cloud-2.asc`) to the machine that you want to add to the cluster.
### Provision the new machine
Switch to the machine that you want to add and proceed with the provisioning procedure described below.
#### Install the snap
First, install the `openstack` snap:
```text
sudo snap install openstack
```
This will install the latest stable version by default. You can use the `--channel` switch to install a different version of OpenStack instead. All machines in the cluster must have the same version of OpenStack installed.
#### Prepare the machine
To prepare the machine for Canonical OpenStack usage, execute the following command:
```text
sunbeam prepare-node-script | bash -x && newgrp snap_daemon
```
This command will:
* ensure all required software dependencies are installed, including the `openssh-server`,
* configure passwordless access to the `sudo` command for all terminal commands for the currently logged in user (i.e. `NOPASSWD:ALL`).
#### Add the machine to the cluster
In order to add the machine to the cluster, execute the `sunbeam cluster join` command on that machine
```text
cat FILE | sunbeam cluster join --role ROLES -
```
`FILE` is a name of the file with the registration token.
`ROLES` is a comma-separated list of roles (`control`, `compute`, `network`, `storage`) to assign to the machine being added.
For example, to add the *cloud-2* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section, execute the following command:
```text
cat cloud-2.asc | sunbeam cluster join --role control,compute,storage -
```
One finished, you should be able to see the following message on your screen:
```text
Node joined cluster with roles: storage, control, compute
```
### Resize the cluster
When provisioning new machines with the `control` role assigned, the cluster needs to be resized to make use of those machines for the purpose of hosting control functions.
To resize the cluster, execute the following command on any of the machines:
```text
sunbeam cluster resize
```
## Scaling the cluster out using Canonical MAAS
The following section provides instructions on scaling the cluster out with Canonical MAAS.
### Requirements
You will need:
* Canonical OpenStack already installed using [Canonical MAAS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-canonical-maas.md)
* one dedicated physical machine:
* with hardware specifications matching minimum hardware specifications as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* ready to be used by MAAS (enlisted, commissioned, configured and tagged)
If you can’t provide an unlimited access to the Internet, see the [Manage a proxied environment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-a-proxied-environment.md) section.
### Provision the new machine
To provision the machine, execute the following command on the first *Sunbeam Client* machine (aka primary node):
```text
sunbeam cluster deploy
```
# index.html.md
# Cluster upgrades
## Overview
A full cluster refresh is a multi-step process. Some components require
dedicated refresh commands and must be refreshed in a specific order before
running the general `sunbeam cluster refresh` command.
#### NOTE
Refreshing across release tracks is not supported. For example, you cannot
use the refresh command to upgrade from `2024.1/stable` to `2025.1/stable`.
## Prerequisites
To ensure the latest updates are available to the cluster charms, refresh the
`openstack` snap before running any cluster refresh commands.
* **Manual provider:** Run `sudo snap refresh openstack` on all nodes.
* **MAAS provider:** Run it on the sunbeam client node only.
#### IMPORTANT
Refreshing the `openstack` snap does not automatically refresh the
cluster. You must explicitly run the dedicated cluster refresh commands
described below to apply updates to the running services.
```
Use `--force` to skip the confirmation prompt:
```text
sunbeam cluster refresh --force
```
Use the `--clear-manifest` flag to remove a previously
stored manifest:
```text
sunbeam cluster refresh --clear-manifest
```
=true
```
Replace `` with the gate key of the feature you want to
enable. For example, to enable the `multi-region` feature gate:
```text
sudo snap set openstack feature.multi-region=true
```
#### NOTE
Experimental features may change or be removed in future releases.
Enable them only in environments where instability is acceptable.
## Disable an experimental feature
To disable a previously enabled experimental feature:
```text
sudo snap set openstack feature.=false
```
# index.html.md
# Enable and deploy a gated storage backend
Use this procedure to unlock a gated in-tree storage backend in the CLI and
then deploy it. For general information about feature gates, see
[Manage experimental features](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/manage-experimental-features.md).
#### NOTE
Pure Storage is generally available and does not require a feature gate.
Add it directly with `sunbeam storage add purestorage ...`.
## List the available feature gates
List the gates that are available in your deployment:
```text
sunbeam list-feature-gates
```
Identify the gate key for the storage backend that you want to deploy. Current
gated in-tree storage backends use keys such as
`feature.storage.dellsc` and `feature.storage.hitachi`.
## Enable the storage backend gate
Unlock the backend by setting its feature gate to `true`:
```text
sudo snap set openstack feature.storage.=true
```
Replace `` with the storage backend name, for example
`dellsc` or `hitachi`.
#### NOTE
Unlocking the gate makes the backend visible in the CLI. It does not deploy
the backend.
## Verify that the backend is unlocked
Run the feature gate command again and confirm that the **Unlocked** column is
set for your storage backend:
```text
sunbeam list-feature-gates
```
If the backend does not appear immediately in the CLI, start a new command
invocation and check again.
In local multi-node deployments, gate changes propagate automatically across
nodes in roughly 5 to 10 seconds. In MAAS deployments, you may need to run the
same `snap set` command on each node even though the gate state is still
stored in the cluster database.
## Review the backend options in the CLI
After the gate is unlocked, confirm that the backend is now exposed by
the storage commands:
```text
sunbeam storage add --help
```
or:
```text
sunbeam storage options
```
Use `sunbeam storage options ` to review the configuration fields
required by the backend before you create its YAML configuration file.
## Deploy the backend
Add the backend by using the backend type, an instance name, and a backend
configuration file:
```text
sunbeam storage add --config-file .yaml
```
For example, to deploy a Hitachi backend:
```text
sunbeam storage add hitachi hitachi-prod --config-file hitachi.yaml
```
## Verify the deployment
List deployed storage backends and confirm that the new backend is present:
```text
sunbeam storage list
```
Once deployed, the backend remains managed separately from the feature gate
state.
# index.html.md
# Deploy a Pure Storage backend
## Overview
Use this procedure to deploy a Pure Storage backend for Cinder. The backend is
deployed as the `cinder-volume-purestorage` charm.
## Requirements
You will need:
* a bootstrapped Canonical OpenStack deployment with storage capability already
in place
* network connectivity from the storage nodes to the Pure Storage array
* a valid Pure Storage API token
* a backend instance name that satisfies Juju application naming rules,
for example `pure-prod`
## Inspect the available options
If you want to review the supported configuration keys before deploying the
backend, run:
```text
sunbeam storage options purestorage
```
## Create the backend configuration
You can provide the backend settings in a YAML file or pass the equivalent CLI
options directly to the deployment command. The required keys are `san-ip`
and `pure-api-token`.
For example, create a file named `purestorage.yaml` with the following
content:
```yaml
san-ip: 192.0.2.10
pure-api-token: 01234567-89ab-cdef-0123-456789abcdef
protocol: iscsi
volume-backend-name: pure-iscsi
backend-availability-zone: az1
pure-iscsi-cidr: 192.0.2.0/24
```
Set `protocol` to `iscsi`, `fc`, or `nvme` to match your deployment.
For NVMe/TCP deployments, you can also set `pure-nvme-cidr` and
`pure-nvme-transport`. Set `pure-nvme-transport` to `tcp`.
## Deploy the backend
Deploy the backend with the backend type (`purestorage`), a
Juju-compatible backend instance name, and the configuration file:
```text
sunbeam storage add purestorage pure-prod --config-file purestorage.yaml
```
If you prefer not to use a file, pass the equivalent options directly on the
command line.
## Verify the backend
Check that the backend has been added:
```text
sunbeam storage list
```
To inspect the deployed backend in more detail, run:
```text
sunbeam storage show pure-prod
```
# index.html.md
# Maintenance mode
## Overview
Maintenance mode helps by protecting the cluster from potentially disruptive maintenance operations. It is useful for performing maintenance tasks on a node that may result in a loss of data or disrupt running services such as firmware upgrades.
Before proceeding, refer to the [Maintenance Mode](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/maintenance-mode.md) to understand its functionality and impact.
## Enabling Maintenance feature
```text
sunbeam enable maintenance
```
Maintenance mode relies on [OpenStack Watcher](https://wiki.openstack.org/wiki/Watcher) to manage hypervisor services and virtual machine instances. Enabling Maintenance mode feature will also enable resource optimization feature to deploy required applications like watcher.
## Usage
### Enabling Maintenance Mode
Before enabling maintenance mode, perform a dry run to check for potential issues:
```text
sunbeam cluster maintenance enable [--disable-migration[=live|cold|both]] --dry-run
Continue to run operations to enable maintenance mode for :
0: change_nova_service_state state=disabled resource=
1: Migrate instance type=live resource=test-vm1
2: Migrate instance type=live resource=test-vm2
3: set-noout-ops
4: assert-noout-flag-set-ops
```
If no issues are reported, enable maintenance mode:
```text
sunbeam cluster maintenance enable [--disable-migration[=live|cold|both]]
Continue to run operations to enable maintenance mode for :
0: change_nova_service_state state=disabled resource=
1: Migrate instance type=live resource=test-vm1
2: Migrate instance type=live resource=test-vm2
3: set-noout-ops
4: assert-noout-flag-set-ops
[y/n]: y
Operation result:
0: change_nova_service_state state=disabled resource= SUCCEEDED
1: Migrate instance type=live resource=test-vm1 SUCCEEDED
2: Migrate instance type=live resource=test-vm2 SUCCEEDED
3: set-noout-ops SUCCEEDED
4: assert-noout-flag-set-ops SUCCEEDED
Enable maintenance for node:
```
#### Controlling migration behavior
By default, enabling maintenance mode will live migrate active instances and cold
migrate inactive instances. The `--disable-migration` flag allows operators to
control this behavior during maintenance.
#### NOTE
If `--disable-migration` is not specified, the default behavior is unchanged.
To disable live migration (cold migrate both active and inactive instances):
```text
sunbeam cluster maintenance enable --disable-migration=live
```
To disable cold migration (live migrate active instances, ignore inactive instances):
```text
sunbeam cluster maintenance enable --disable-migration=cold
```
To disable all migration (only stop active instances, ignore inactive instances):
```text
sunbeam cluster maintenance enable --disable-migration=both
```
Or equivalently, without specifying a value:
```text
sunbeam cluster maintenance enable --disable-migration
```
The `--disable-migration` flag can be combined with `--dry-run` to preview the
effect before applying:
```text
sunbeam cluster maintenance enable --disable-migration=live --dry-run
```
### Disabling Maintenance Mode
To disable maintenance mode, first run a dry run to validate the operation:
```text
sunbeam cluster maintenance disable --dry-run
required operations to disable maintenance mode for :
0: EnableHypervisorStep
1: unset-noout-ops
2: assert-noout-flag-unset-ops
3: start-osd-ops
Disable maintenance for node:
```
If the output confirms a safe transition, disable maintenance mode:
```text
sunbeam cluster maintenance disable
Continue to run operations to disable maintenance mode for :
0: EnableHypervisorStep
1: unset-noout-ops
2: assert-noout-flag-unset-ops
3: start-osd-ops
[y/n]: y
Operation result:
0: EnableHypervisorStep SUCCEEDED
1: unset-noout-ops SUCCEEDED
2: assert-noout-flag-unset-ops SUCCEEDED
3: start-osd-ops SUCCEEDED
Disable maintenance for node:
```
# index.html.md
# Scaling the cluster in
Canonical OpenStack scales in, meaning that you can remove machines from the cluster if you no longer need them.
## Scaling the cluster in using the manual bare metal provider
The following section provides instructions on scaling the cluster in with the manual bare metal provider.
These instructions apply to all node types but the primary node. For instruction on the latter, refer to [Removing the primary node](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/removing-the-primary-node.md) section of this documentation.
### Remove the machine from the cluster
To remove the machine from the cluster, execute the `sunbeam cluster remove` command on the primary node:
```text
sunbeam cluster remove FQDN
```
`FQDN` is a fully qualified domain name (FQDN) of the machine being removed.
For example, to remove the *cloud-2* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section, execute the following command:
```text
sunbeam cluster remove cloud-2.example.com
```
### Remove components from the machine
Software components now need to be removed from the target node. Perform all the below steps on the target node.
Remove the Juju agent:
```text
sudo /sbin/remove-juju-services
```
Remove the `juju` snap:
```text
sudo snap remove --purge juju
```
Remove Juju configuration:
```text
rm -rf ~/.local/share/juju
```
Remove the `openstack-hypervisor` and `openstack` snaps:
```text
sudo snap remove --purge openstack-hypervisor
sudo snap remove --purge openstack
```
Remove `openstack` snap configuration:
```text
rm -rf ~/.local/share/openstack
```
Remove the `k8s` snap:
```text
sudo k8s remove-node
sudo snap remove --purge k8s
```
#### NOTE
The above steps can take a few minutes to complete.
Remove the disk(s) used by MicroCeph on this node:
```text
sudo microceph disk list
sudo microceph disk remove
```
Remove the `microceph` snap:
```text
sudo microceph disk list
sudo snap remove --purge microceph
```
If required clean the disk(s) identified in the earlier command:
#### WARNING
The `dd` command will result in the permanent erasure of data. It is vital that you have specified the correct disk path to avoid unintended data loss.
```text
sudo dd if=/dev/zero of=PATH bs=4M count=10
```
`PATH` is a path to the disk being cleaned.
Clear the remaining network configuration with a reboot:
```text
sudo reboot
```
## Scaling the cluster in using Canonical MAAS
The following section provides instructions on scaling the cluster out with Canonical MAAS.
Coming soon.
# index.html.md
# Removing the primary node
Removing the primary node refers to the removal of the first (bootstrap) node in the cluster and is only applicable when using the manual bare metal provider.
If the deployment consists of multiple nodes then remove all non-primary nodes before removing the primary node. Refer to the [Scaling the cluster in](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-in.md) section of this documentation for exact instructions on how to do that.
## Remove components from the machine
#### WARNING
Removing the primary node will destroy the entire Canonical OpenStack deployment.
Software components now need to be removed from the primary node. Perform all the below steps on the primary node.
Remove the Juju models:
```text
juju destroy-model --destroy-storage --no-prompt --force --no-wait openstack
juju destroy-model --destroy-storage --no-prompt --force --no-wait admin/openstack-machines
```
Remove the Juju controller:
```text
juju destroy-controller --no-prompt --destroy-storage --force --no-wait localhost-localhost
```
Remove the Juju agent:
```text
sudo /sbin/remove-juju-services
```
Remove the `juju` snap:
```text
sudo snap remove --purge juju
```
Remove Juju configuration:
```text
rm -rf ~/.local/share/juju
sudo rm -rf /var/lib/juju/dqlite
sudo rm -rf /var/lib/juju/system-identity
sudo rm -rf /var/lib/juju/bootstrap-params
```
Remove the `openstack-hypervisor` and `openstack` snaps:
```text
sudo snap remove --purge openstack-hypervisor
sudo snap remove --purge openstack
```
Remove `openstack` snap configuration:
```text
rm -rf ~/.local/share/openstack
```
Remove the `k8s` snap:
```text
sudo snap remove --purge k8s
```
Remove the `microovn` snap:
```text
sudo snap remove --purge microovn
```
#### NOTE
The above steps can take a few minutes to complete.
Remove the disk(s) used by MicroCeph on this node:
```text
sudo microceph disk list
sudo microceph disk remove --bypass-safety-checks
```
#### NOTE
`sudo microceph disk list` may list the un-partitioned disks on the system. These can be ignored.
Remove the `microceph` snap:
```text
sudo snap remove --purge microceph
```
If required clean the disk(s) identified in the earlier command:
#### WARNING
The `dd` command will result in the permanent erasure of data. It is vital that you have specified the correct disk path to avoid unintended data loss.
```text
sudo dd if=/dev/zero of=PATH bs=4M count=10
```
`PATH` is a path to the disk being cleaned.
Clear the remaining network configuration with a reboot:
```text
sudo reboot
```
# index.html.md
# Troubleshooting
* [Inspecting the cluster](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#overview)
* [Juju](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#juju)
* [OpenStack Hypervisor](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#openstack-hypervisor)
* [Canonical Kubernetes](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#canonical-kubernetes)
* [Services hosted on Canonical Kubernetes](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#services-hosted-on-canonical-kubernetes)
* [MicroCeph](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#microceph)
* [Sunbeam MicroCluster](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#sunbeam-microcluster)
* [Terraform plans](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/troubleshooting/inspecting-the-cluster.md#terraform-plans)
# index.html.md
# Inspecting the cluster
## Overview
Sunbeam aims to remove the need for an operator to know all of the
technical detail about how to deploy an OpenStack cloud; however when
something does go wrong it’s important to be able to inspect the various
components in order to discover the nature of the problem.
## Juju
Sunbeam makes extensive use of Juju to manage the components of the
OpenStack cloud across both the underlying nodes and on Kubernetes (see
Canonical Kubernetes).
Sunbeam uses two Juju models for managing the various components
deployed to create the OpenStack cloud.
### Juju controller authentication
Sunbeam uses a set of credentials for each node in the cluster for
access to the Juju controller. The authenticated session for each node
expires after 24 hours so to use the `juju` command directly it may be
necessary to re-authenticate.
Juju commands will prompt for a password once the session has expired -
the password for the node’s user can be found in
`${HOME}/snap/openstack/current/account.yaml`.
Alternatively the `juju-login` helper can be used to re-authenticate
with the Juju controller:
```default
sunbeam utils juju-login
```
### Controller model
The `controller` model contains the Canonical OpenStack components that are
placed directly on the nodes that make up the deployment. The status of
this model can be queried using the following command:
```default
juju status -m admin/controller
```
This should work from any node in the deployment
This model contains the application deployments for the Canonical K8s
(control role), MicroOVN (network role), MicroCeph (storage role) and OpenStack Hypervisor
(compute role) components of Canonical OpenStack.
Depending on the roles assigned to individual machines, a unit of each
of the applications should be present in the model.
The controller application is a special application that represents the
Juju controller.
### OpenStack model
The `openstack` model contains all of the components of the OpenStack
Cloud that are deployed on top of Kubernetes (provided by Canonical K8s
from the `controller` model).
The status of this model can be queried using the following command:
```default
juju status -m openstack
```
This should work from any node in the deployment.
#### CAUTION
If the storage role is not specified for any nodes in the deployment the
`cinder-ceph` application will remain in a blocked state. This is expected
and means that the deployed OpenStack cloud does not support the block storage
service.
## OpenStack Hypervisor
The OpenStack Hypervisor is a snap based component that provides all of
the core functionality needed to operate a hypervisor as part of an
OpenStack Cloud. This include Nova Compute, Libvirt+QEMU for hardware
based virtualization, OVN and OVS for software defined networking and
supporting services to provide metadata to instances.
The status of the snap’s services can be checked using:
```default
sudo systemctl status snap.openstack-hypervisor.*
```
All log output for the services can be captured by consulting the
journal:
```default
sudo journalctl -xe -u snap.openstack-hypervisor.*
```
This component is deployed and integrated into the cloud using the
`openstack-hypervisor` charm that is deployed in the `controller`
model.
## Canonical Kubernetes
Canonical Kubernetes (K8s) provides Kubernetes as part of Canonical OpenStack.
The current status of the K8s cluster can be checked by running:
```default
sudo k8s status
```
A more in-depth inspection and generation of a archive suitable for use
as part of a bug submission can also be completed by running:
```default
sudo k8s inspect
```
## Services hosted on Canonical Kubernetes
Components of OpenStack Control Plane are hosted on K8S.
You can get the different units by running:
```default
sudo k8s kubectl get pods --namespace openstack
```
If a pod is in an error state, or is stuck in a `Pending` state, you
can retrieve more information on it and events related to it by running:
```default
sudo k8s kubectl describe --namespace openstack pod
```
To fetch the logs of a specific unit on K8S, it is necessary to
know the name of the containers running inside a given pod. To get the
names of the containers:
```default
sudo k8s kubectl get pod --namespace openstack -o jsonpath="{.spec.containers[*].name}"
```
A Juju unit will always have a `charm` container running the Juju
agent responsible for running the charm. To fetch logs associated with
the charm of a particular unit:
```default
sudo k8s kubectl logs --namespace openstack --container charm
```
#### NOTE
The charm container logs are also available through `juju debug-log -m openstack`,
and will be present in the sunbeam inspection report.
To fetch the payload logs, use:
```default
sudo k8s kubectl logs --namespace openstack --container
```
## MicroCeph
If nodes are deployed with the storage role enabled, MicroCeph will be
deployed as part of the cluster.
The status of MicroCeph can be checked using:
```default
sudo microceph status
```
and the status of the Ceph cluster can be displayed using:
```default
sudo ceph -s
```
## Sunbeam MicroCluster
Sunbeam MicroCluster provides some basic cluster coordination and state
sharing services as part of Canonical OpenStack. The status of the nodes
participating in the Sunbeam MicroCluster can be queried using the
following command:
```default
sunbeam cluster list
```
The state of the local daemon managing the nodes participation in the
cluster can also be checked and the log output captured if need be:
```text
sudo systemctl status snap.openstack.clusterd.service
sudo journalctl -xe -u snap.openstack.clusterd.service
```
## Terraform plans
Sunbeam makes extensive use of Terraform to deploy OpenStack. In some
rare cases a Terraform plan can stay locked making it impossible to
re-run commands on the bootstrap node or add new nodes to the
deployment.
To list the current lock state of all Terraform plans:
```default
sunbeam plans list
```
To unlock a specific Terraform plan:
```default
sunbeam plans unlock
```
This command may prompt you to confirm unlocking depending on how recent
the lock timestamp is.
#### CAUTION
Ensure that there are no administrative operations underway in the
deployment when unlocking a Terraform plan. Otherwise, the deployment’s
integrity can be compromised.
Logs from running Terraform can be found in:
```default
$HOME/snap/openstack/common/etc//*/terraform-*.log
```
#### NOTE
cloud_name is the name of the juju cloud created when bootstrapping sunbeam.
It is usually the only directory in $HOME/snap/openstack/common/etc/.
# index.html.md
# Telemetry
This feature deploys the OpenStack Telemetry services Ceilometer, Aodh,
Gnocchi, and OpenStack Exporter.
## Enabling Telemetry
To enable Telemetry, run the following command:
```bash
sunbeam enable telemetry
```
Use the OpenStack CLI to create and manage alarms. See the upstream
[Aodh
documentation](https://docs.openstack.org/aodh/latest/admin/telemetry-alarms.html#using-alarms)
for details.
## Disabling Telemetry
To disable Telemetry, run the following command:
```bash
sunbeam disable telemetry
```
This will terminate the application but not remove it from the model. To
do that, run the following:
```default
juju remove-application --force --no-wait --no-prompt -m openstack \
ceilometer gnocchi gnocchi-mysql-router aodh aodh-mysql-router
```
## Usage
### Alarms
Users need the role `admin` to be able to manage alarms.
Create alarm `memory_high` with metric `metric.usage` and alarm
action `log` using the following command:
```default
openstack alarm create \
--name memory_high \
--type gnocchi_resources_threshold \
--description 'instance consuming memory' \
--metric memory.usage \
--threshold 2000 \
--comparison-operator gt \
--aggregation-method mean \
--granularity 300 \
--evaluation-periods 3 \
--alarm-action 'log://' \
--resource-id \
--resource-type instance
```
Sample output:
```default
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| aggregation_method | mean |
| alarm_actions | ['log:'] |
| alarm_id | d365506b-fc14-479d-b34d-0f3ae267a858 |
| comparison_operator | gt |
| description | instance consuming memory |
| enabled | True |
| evaluate_timestamp | 2023-10-13T04:07:14.849164 |
| evaluation_periods | 3 |
| granularity | 300 |
| insufficient_data_actions | [] |
| metric | memory.usage |
| name | memory_high |
| ok_actions | [] |
| project_id | 815325ab42e443fbb6fc6eb8905c5aa8 |
| repeat_actions | False |
| resource_id | 1f12876c-b320-436a-9ae9-5fd8e065e69f |
| resource_type | instance |
| severity | low |
| state | insufficient data |
| state_reason | Not evaluated yet |
| state_timestamp | 2023-10-13T04:07:14.792561 |
| threshold | 2000.0 |
| time_constraints | [] |
| timestamp | 2023-10-13T04:07:14.792561 |
| type | gnocchi_resources_threshold |
| user_id | d9730bd835bc4620ab3e6b06c5b17477 |
+---------------------------+--------------------------------------+
```
Check the metrics for `memory.usage` using the following command:
```default
openstack metric measures show -r memory.usage --granularity 300
```
Sample output:
```default
+---------------------------+-------------+---------------+
| timestamp | granularity | value |
+---------------------------+-------------+---------------+
| 2023-10-13T03:45:00+00:00 | 300.0 | 2138.51171875 |
| 2023-10-13T03:50:00+00:00 | 300.0 | 2138.46875 |
| 2023-10-13T03:55:00+00:00 | 300.0 | 2138.4609375 |
| 2023-10-13T04:00:00+00:00 | 300.0 | 2138.4609375 |
| 2023-10-13T04:05:00+00:00 | 300.0 | 2138.4609375 |
+---------------------------+-------------+---------------+
```
Check alarm history for any alarm events triggered using the following
command:
```default
openstack alarm-history show --fit-width
```
Sample output:
```default
+----------------------------+------------------+-------------------------------------------------------------------------------------------------------------------+--------------------------------------+
| timestamp | type | detail | event_id |
+----------------------------+------------------+-------------------------------------------------------------------------------------------------------------------+--------------------------------------+
| 2023-10-13T04:08:45.607387 | state transition | {"state": "alarm", "transition_reason": "Transition to alarm due to 3 samples outside threshold, most recent: | 1ae90db8-124a-4e9a-9e76-5a81216ac54c |
| | | 2138.4609375"} | |
| 2023-10-13T04:07:14.792561 | creation | {"alarm_id": "d365506b-fc14-479d-b34d-0f3ae267a858", "type": "gnocchi_resources_threshold", "enabled": true, | b991407b-511e-40ba-a353-f63e8e60782c |
| | | "name": "memory_high", "description": "instance consuming memory", "timestamp": "2023-10-13T04:07:14.792561", | |
| | | "user_id": "d9730bd835bc4620ab3e6b06c5b17477", "project_id": "815325ab42e443fbb6fc6eb8905c5aa8", "state": | |
| | | "insufficient data", "state_timestamp": "2023-10-13T04:07:14.792561", "state_reason": "Not evaluated yet", | |
| | | "ok_actions": [], "alarm_actions": ["log://"], "insufficient_data_actions": [], "repeat_actions": false, | |
| | | "time_constraints": [], "severity": "low", "rule": {"granularity": 300, "comparison_operator": "gt", "threshold": | |
| | | 2000.0, "aggregation_method": "mean", "evaluation_periods": 3, "metric": "memory.usage", "resource_id": | |
| | | "1f12876c-b320-436a-9ae9-5fd8e065e69f", "resource_type": "instance"}} | |
+----------------------------+------------------+-------------------------------------------------------------------------------------------------------------------+--------------------------------------+
```
Alarm `memory_high` is created with alarm action `log`, so check for
log events on `aodh-evaluator`:
```default
sudo k8s kubectl -n openstack logs aodh-0 -c aodh-notifier | grep memory_high
2023-10-13T04:08:45.650Z [aodh-notifier] 2023-10-13 04:08:45.648 17 INFO aodh.notifier.log [-]
Notifying alarm memory_high d365506b-fc14-479d-b34d-0f3ae267a858 of low priority from insufficient data to alarm with action log: because Transition to alarm due to 3 samples outside threshold, most recent: 2138.4609375.
```
### OpenStack Exporter
When the [Observability](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md) feature is enabled, you’ll have
access to the Grafana OpenStack dashboards, providing insights about the
cloud usage.
- OpenStack Dashboard: an overview of the various OpenStack components
- OpenStack Overview: higher level overview of the OpenStack deployment
- OpenStack Hypervisor Overview: detailed information of the
hypervisors
# index.html.md
# Optional Features
* [Baremetal as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md)
* [Enabling Baremetal](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md#enabling-baremetal)
* [Managing nova-ironic shards](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md#managing-nova-ironic-shards)
* [Managing Ironic Conductor groups](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md#managing-ironic-conductor-groups)
* [Managing Neutron Switch Configurations](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md#managing-neutron-switch-configurations)
* [Disabling Baremetal](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/baremetal.md#disabling-baremetal)
* [Containers as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/caas.md)
* [Enabling CaaS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/caas.md#enabling-caas)
* [Disabling CaaS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/caas.md#disabling-caas)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/caas.md#usage)
* [Limitations:](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/caas.md#limitations)
* [DNS as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/dns.md)
* [Enabling DNS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/dns.md#enabling-dns)
* [Disabling DNS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/dns.md#disabling-dns)
* [Fetching DNS service address](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/dns.md#fetching-dns-service-address)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/dns.md#usage)
* [Images Sync](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/images-sync.md)
* [Enable Images Sync](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/images-sync.md#enable-images-sync)
* [Disable Images Sync](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/images-sync.md#disable-images-sync)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/images-sync.md#usage)
* [Instance Recovery](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md)
* [Enabling Instance Recovery](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#enabling-instance-recovery)
* [Disabling Instance Recovery](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#disabling-instance-recovery)
* [Encrypted disk recovery](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#encrypted-disk-recovery)
* [Instance Evacuation Recovery methods](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#instance-evacuation-recovery-methods)
* [Create a failover segment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#create-a-failover-segment)
* [Supplementary information](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#supplementary-information)
* [Limitations](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md#limitations)
* [LDAP Integration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ldap.md)
* [Enabling LDAP](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ldap.md#enabling-ldap)
* [Disabling LDAP](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ldap.md#disabling-ldap)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ldap.md#usage)
* [Load Balancer as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md)
* [Enabling Load Balancer](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md#enabling-load-balancer)
* [Enabling the Amphora provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md#enabling-the-amphora-provider)
* [Disabling Load Balancer](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md#disabling-load-balancer)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md#usage)
# Managing TLS
* [Managing TLS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/index.md)
* [Implementing TLS using a third-party CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md)
* [Managing CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md)
* [Managing Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md)
* [Object Storage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/object-storage.md)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/object-storage.md#usage)
* [Observability](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md)
* [Connect to an existing COS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md#connect-to-an-existing-cos)
* [Deploy COS in Canonical OpenStack](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md#deploy-cos-in-canonical-openstack)
* [Login Grafana dashboard](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md#login-grafana-dashboard)
* [Dashboard](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md#dashboard)
* [Orchestration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/orchestration.md)
* [Enabling Orchestration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/orchestration.md#enabling-orchestration)
* [Disabling Orchestration](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/orchestration.md#disabling-orchestration)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/orchestration.md#usage)
* [Resource Optimization](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/resource-optimization.md)
* [Enabling Resource Optimization](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/resource-optimization.md#enabling-resource-optimization)
* [Disabling Resource Optimization](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/resource-optimization.md#disabling-resource-optimization)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/resource-optimization.md#usage)
* [Limitations](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/resource-optimization.md#limitations)
* [Secrets as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md)
* [Enabling Secrets](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#enabling-secrets)
* [Disabling Secrets](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#disabling-secrets)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#usage)
* [Audit secret decrypt access](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#audit-secret-decrypt-access)
* [Remove secret decrypt access](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#remove-secret-decrypt-access)
* [Shared Filesystems as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/shared-filesystem.md)
* [Enabling Shared Filesystems](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/shared-filesystem.md#enabling-shared-filesystems)
* [Disabling Shared Filesystems](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/shared-filesystem.md#disabling-shared-filesystems)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/shared-filesystem.md#usage)
* [Telemetry](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/telemetry.md)
* [Enabling Telemetry](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/telemetry.md#enabling-telemetry)
* [Disabling Telemetry](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/telemetry.md#disabling-telemetry)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/telemetry.md#usage)
* [Ubuntu Pro](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ubuntu-pro.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ubuntu-pro.md#overview)
* [Enabling Ubuntu Pro](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ubuntu-pro.md#enabling-ubuntu-pro)
* [Disabling Ubuntu Pro](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ubuntu-pro.md#disabling-ubuntu-pro)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/ubuntu-pro.md#usage)
* [Validation](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/validation.md)
* [Overview](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/validation.md#overview)
* [Enable Validation](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/validation.md#enable-validation)
* [Disable Validation](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/validation.md#disable-validation)
* [Usage](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/validation.md#usage)
* [Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md)
* [Enabling Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#enabling-vault)
* [Initializing Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#initializing-vault)
* [Unsealing Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#unsealing-vault)
* [Authorizing Vault charm](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#authorizing-vault-charm)
* [Vault status](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#vault-status)
* [Disabling Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md#disabling-vault)
# index.html.md
# Instance Recovery
This feature deploys [Masakari](https://docs.openstack.org/masakari/latest/index.html), the OpenStack Instance High Availability service. This feature
requires at least two compute nodes in the deployment.
## Enabling Instance Recovery
To enable Instance Recovery, run the following command:
```bash
sunbeam enable instance-recovery
```
#### NOTE
For MAAS provider, ensure to reserve an IP on storage subnet with label -storage-ippool,
see [reference](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-canonical-maas.md#reserved-ipranges).
This will be used to perform health checks over the storage network.
When [Secrets](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md) is enabled, Masakari is granted
permission to decrypt Barbican secrets by default. This allows Masakari to
rebuild VMs with encrypted disks during instance recovery.
## Disabling Instance Recovery
To disable Instance Recovery, run the following command:
```bash
sunbeam disable instance-recovery
```
## Encrypted disk recovery
Masakari needs access to Barbican secret payloads to recover VMs that use
encrypted disks. Canonical OpenStack enables this access by default with the
`enable-secret-access` option on the `masakari-k8s` charm.
To opt out durably, set the option in the deployment manifest:
```yaml
software:
charms:
masakari-k8s:
config:
enable-secret-access: false
```
Apply the updated manifest when enabling, or re-enabling, Instance Recovery:
```default
sunbeam enable --manifest instance-recovery
```
Disabling this option stops Masakari from requesting the `secret-decrypter`
role. It does not remove existing Keystone role assignments. To audit or remove
existing assignments, follow [the Secrets role audit guidance](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md#audit-secret-decrypt-access).
## Instance Evacuation Recovery methods
With Masakari, compute nodes are grouped into failover segments. In the event of a compute node
failure, that node’s instances are moved onto another compute node within the same segment.
The destination node is determined by the recovery method configured for the affected segment.
There are four methods:
* reserved_host
: The `reserved_host` recovery method relocates instances to a subset of non-active nodes.
Because these nodes are not active and are typically resourced adequately for failover duty,
there is a guarantee that sufficient resources will exist on a reserved node to accommodate
migrated instances.
* auto
: The `auto` recovery method relocates instances to any available node in the same segment.
Because all the nodes are active, contrarily to the `reserved_host` method, there is no
guarantee that sufficient resources will exist on the destination node to accommodate migrated
instances.
* rh_priority
: Attempts to evacuate instances using the `reserved_host` method. If the latter is unsuccessful
the auto method will be used.
* auto_priority
: Attempts to evacuate instances using the `auto` method. If the latter is unsuccessful the
`reserved_host` method will be used.
## Create a failover segment
Run the following command to create a failover segment
```default
openstack segment create NAME RECOVERY_METHOD SERVICE_TYPE
```
For example, to create a service S1 with `reserved_host` for service type COMPUTE
```default
openstack segment create S1 reserved_host COMPUTE
```
Sample output for the above command:
```default
+-----------------+--------------------------------------+
| Field | Value |
+-----------------+--------------------------------------+
| created_at | 2024-10-18T04:32:21.000000 |
| updated_at | None |
| uuid | 7980f98d-8458-467b-a5cd-ceb5e6b94cdb |
| name | S1 |
| description | None |
| id | 1 |
| service_type | COMPUTE |
| recovery_method | reserved_host |
| is_enabled | True |
+-----------------+--------------------------------------+
```
Add hosts to failover segment
Run the following command to add host to a failover segment
```default
openstack segment host create NAME TYPE CONTROL_ATTRIBUTE SEGMENT [--reserved] [--on-maintenance]
```
For example, to add host sunbeam1.maas to segment S1, run the following command:
```default
openstack segment host create sunbeam1.maas COMPUTE SSH S1
```
Sample output of the above command:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2024-10-18T04:32:56.000000 |
| updated_at | None |
| uuid | 7cd21df7-ab9e-4bc3-a69b-13df4f53b125 |
| name | sunbeam1.maas |
| type | COMPUTE |
| control_attributes | SSH |
| reserved | False |
| on_maintenance | False |
| failover_segment_id | 7980f98d-8458-467b-a5cd-ceb5e6b94cdb |
+---------------------+--------------------------------------+
```
Hosts can be reserved when adding to failover segment. These hosts will be used in
`reserved_host` recovery method. For example, to add host `sunbeam3.maas` as reserved,
run the following command:
```default
openstack segment host create --reserved True sunbeam3.maas COMPUTE SSH S1
```
Sample output of the above command:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2024-10-18T04:33:44.000000 |
| updated_at | None |
| uuid | a641ce79-3610-4753-b59f-9902ee8da4c3 |
| name | sunbeam3.maas |
| type | COMPUTE |
| control_attributes | SSH |
| reserved | True |
| on_maintenance | False |
| failover_segment_id | 7980f98d-8458-467b-a5cd-ceb5e6b94cdb |
+---------------------+--------------------------------------+
```
### List hosts in failover segment
Run below command to list hosts in a failover segment:
```default
openstack segment host list S1 -c name -c reserved -c on_maintenance
```
Sample output of the above command:
```default
+---------------+----------+----------------+
| name | reserved | on_maintenance |
+---------------+----------+----------------+
| sunbeam3.maas | True | False |
| sunbeam2.maas | False | False |
| sunbeam1.maas | False | False |
+---------------+----------+----------------+
```
### Verify reserved_host recovery method
Create failover segment and add hosts to the failover segment. Ensure add few hosts with
`--reserved` flag. Once this is done, disable the reserved host using the following command:
```default
openstack compute service set --disable HOSTNAME nova-compute
```
Run the following command to check status of nova-compute services:
```default
openstack compute service list -c Host -c Status -c State --service nova-compute
```
Sample output of the above command:
```default
+---------------+----------+-------+
| Host | Status | State |
+---------------+----------+-------+
| sunbeam1.maas | enabled | up |
| sunbeam2.maas | enabled | up |
| sunbeam3.maas | disabled | up |
+---------------+----------+-------+
```
In the above output, observe the state of reserved node `sunbeam3.maas` is disabled.
Check the instances running in the cloud using command
```default
openstack server list --long --all-projects -c Name -c Host
```
Sample output:
```default
+----------+---------------+
| Name | Host |
+----------+---------------+
| testvm-2 | sunbeam1.maas |
| testvm-1 | sunbeam1.maas |
+----------+---------------+
```
Simulate a compute node failure by shutting down the interface or bringing down the interface. If
the interface is one of the management or data or storage networks, masakari service will trigger
actions based on recovery strategy. The actions that will be taken for each network failure are
as follows:
#### Recovery actions based on network status
| Management Network | Tenant Network | Storage Network | Action |
|----------------------|------------------|-------------------|----------|
| up | up | up | – |
| up | up | down | recovery |
| up | down | up | – |
| up | down | down | recovery |
| down | up | up | – |
| down | up | down | recovery |
| down | down | up | – |
| down | down | down | recovery |
Verify Hosts in Failover segment once the failure is simulated:
```default
openstack segment host list S1 -c name -c reserved -c on_maintenance
```
Sample output after shutting down node ‘sunbeam1.maas’
```default
+---------------+----------+----------------+
| name | reserved | on_maintenance |
+---------------+----------+----------------+
| sunbeam3.maas | False | False |
| sunbeam2.maas | False | False |
| sunbeam1.maas | False | True |
+---------------+----------+----------------+
```
In the above output, observe the failure node `sunbeam1.maas` is changed to maintenance mode and
reserved node `sunbeam3.maas` is no longer reserved.
Verify the compute service status
```default
openstack compute service list -c Host -c Status -c State --service nova-compute
```
Sample output:
```default
+---------------+----------+-------+
| Host | Status | State |
+---------------+----------+-------+
| sunbeam1.maas | disabled | down |
| sunbeam3.maas | enabled | up |
| sunbeam2.maas | enabled | up |
+---------------+----------+-------+
```
In the above output, observe status of failed node is changed to disabled and reserved node is
changed to `enabled`.
Verify if the instances are evacuated using the below command:
```default
openstack server list --long --all-projects -c Name -c Host
```
Sample output showing instances on failed node are now moved to reserved node.
```default
+----------+---------------+
| Name | Host |
+----------+---------------+
| testvm-2 | sunbeam3.maas |
| testvm-1 | sunbeam3.maas |
+----------+---------------+
```
### Verify auto recovery method
Create failover segment and add hosts to the failover segment without any –reserved flag.
Run below command to list hosts in a failover segment:
```default
openstack segment host list S1 -c name -c reserved -c on_maintenance
```
Sample output of the above command:
```default
+---------------+----------+----------------+
| name | reserved | on_maintenance |
+---------------+----------+----------------+
| sunbeam3.maas | False | False |
| sunbeam2.maas | False | False |
| sunbeam1.maas | False | False |
+---------------+----------+----------------+
```
The reserved flag in all nodes is False.
Simulate the compute node failure and observe Verify Hosts in Failover segment :
```default
openstack segment host list S1 -c name -c reserved -c on_maintenance
```
Sample output after shutting down node `sunbeam1.maas`
```default
+---------------+----------+----------------+
| name | reserved | on_maintenance |
+---------------+----------+----------------+
| sunbeam3.maas | False | False |
| sunbeam2.maas | False | False |
| sunbeam1.maas | False | True |
+---------------+----------+----------------+
```
Observe in the above output node `sunbeam1.maas` is in Maintenance mode.
Verify if the instances are evacuated using the below command:
```default
openstack server list --long --all-projects -c Name -c Host
```
Sample output showing instances on failed node are now moved to different compute hosts.
```default
+----------+---------------+
| Name | Host |
+----------+---------------+
| testvm-2 | sunbeam2.maas |
| testvm-1 | sunbeam3.maas |
+----------+---------------+
```
Observe in the above output the instances are scheduled to any available compute nodes.
## Instance Restart
The enabling of the instance restart feature is done on a per-instance basis.
For example, tag instance `testvm-1` as HA-enabled in order to have it restarted automatically on
its hypervisor:
```default
openstack server set --property HA_Enabled=True testvm-1
```
An instance failure can be simulated by killing its process. First determine its hypervisor and
`qemu` guest name:
```default
openstack server show testvm-1 -c OS-EXT-SRV-ATTR:host -c OS-EXT-SRV-ATTR:instance_name
```
Sample output:
```default
+-------------------------------+-------------------+
| Field | Value |
+-------------------------------+-------------------+
| OS-EXT-SRV-ATTR:host | sunbeam3.maas |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
+-------------------------------+-------------------+
```
Check the current PID, kill the process, wait a minute, and verify that a new process gets started:
```default
juju exec --unit openstack-hypervisor/2 'pgrep -f guest=instance-00000001'
juju exec --unit openstack-hypervisor/2 'sudo pkill -f -9 guest=instance-00000001'
juju exec --unit openstack-hyperivsor/2 'pgrep -f guest=instance-00000001'
```
## Supplementary information
This section contains information that can be useful when working with Masakari.
* Once a failed node has been re-inserted into the cloud it will show, in Nova, as `disabled` but
`up` and, in Masakari, as `on_maintenance`. It can become an active hypervisor with:
```default
openstack compute service set --enable HOSTNAME nova-compute
openstack segment host update --on_maintenance=False SEGMENT HOSTNAME
```
* A segment’s recovery method can be updated with:
```default
openstack segment update --recovery_method RECOVERY_METHOD --service_type COMPUTE SEGMENT
```
* A node cannot be assigned to a segment while it’s assigned to another segment. It must first be
removed from the current segment with:
```default
openstack segment host delete SEGMENT HOSTNAME
```
* A node’s reserved status can be updated with:
```default
openstack segment host update --reserved= SEGMENT HOSTNAME
```
* VMs booting from a Cinder volume with `multiattach=False` might not be evacuated within the
Masakari evacuation window. Nova can trigger evacuation while waiting for Cinder to detach the
volume from the source host. In this case, the evacuation remains pending until the original
attachment is released or the source host becomes available again.
As a workaround, fence the failed source host or otherwise confirm that it is powered off and
that the original attachment is no longer active before resetting the Cinder volume state:
```bash
openstack volume set --state available
```
Resetting the volume state earlier can allow the same boot volume to be attached on another host
while it is still in use on the source host, which can corrupt the guest filesystem.
## Limitations
* Recovery actions for Instance recovery methods can detect only Management network failure. For
failure in tenant or storage network, recovery action is triggered by updating the compute
node to maintenance but evacuation of instances does not happen.
# index.html.md
# Vault
This feature deploys Vault, a tool for securely managing secrets used in
modern computing (e.g. passwords, certificates, API keys).
## Enabling Vault
To enable Vault, run the following command:
```default
sunbeam enable vault
```
Vault units will be in blocked state after this step.
## Initializing Vault
To initialize Vault, run the following command:
```default
sunbeam vault init KEY_SHARES KEY_THRESHOLD
```
KEY_SHARES - Number of key shares to be generated by vault
KEY_THRESHOLD - Minimal number of key shares to be used to unseal vault
Output of the above command with 5 key shares and 3 key threshold looks like:
```default
Unseal keys:
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
cccccccccccccccccccccccccccccccccccccccccccc
dddddddddddddddddddddddddddddddddddddddddddd
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
Root token: fff.ffffffffffffffffffffffff
```
It is recommended to store each of the unseal keys and root token into different files
and keep them secure.
## Unsealing Vault
To unseal Vault, run the following command:
```default
cat | sunbeam vault unseal -
```
Unsealing the Vault requires minimum KEY_THRESHOLD keys to be provided to vault.
So the unseal command should be executed KEY_THRESHOLD times. This will unseal
the vault leader unit.
To unseal the non-leader units, repeat the unseal commands again.
For example, the process to unseal the Vault with 3 units, initialized with
5 key shares and 3 key threshold looks like:
Unseal with the first key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status: 2 key shares required to unseal
```
Unseal with the second key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status: 1 key shares required to unseal
```
Unseal with the third key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status: completed for leader unit.
Rerun `sunbeam vault unseal` command to unseal non-leader units.
```
The leader unit gets unsealed and non-leader units are in sealed state.
Now repeat the process to unseal non-leader units.
Unseal with the first key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status:
vault/1 : 2 key shares required to unseal
vault/2 : 2 key shares required to unseal
```
Unseal with the second key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status:
vault/1 : 1 key shares required to unseal
vault/2 : 1 key shares required to unseal
```
Unseal with the third key:
```default
$ cat | sunbeam vault unseal -
Vault unseal operation status: completed
```
Unsealing vault process completed.
## Authorizing Vault charm
To authorize vault charm, run the following command:
```default
$ cat | sunbeam vault authorize-charm -
Vault charm is authorized.
```
After 5 minutes (update-status-interval time), Juju status should show all units as active.
```default
$ juju status -m openstack vault
Model Controller Cloud/Region Version SLA Timestamp
openstack sunbeam-controller immune-drum-k8s/localhost 3.5.4 unsupported 07:12:02Z
SAAS Status Store URL
microceph active local admin/controller.microceph
App Version Status Scale Charm Channel Rev Address Exposed Message
vault active 3 vault-k8s 1.16/stable 280 10.152.183.222 no
Unit Workload Agent Address Ports Message
vault/0* active idle 10.1.183.201
vault/1 active idle 10.1.183.234
vault/2 active idle 10.1.183.235
Offer Application Charm Rev Connected Endpoint Interface Role
cert-distributor keystone keystone-k8s 211 2/2 send-ca-cert certificate_transfer provider
certificate-authority certificate-authority self-signed-certificates 155 1/1 certificates tls-certificates provider
cinder-ceph cinder-ceph cinder-ceph-k8s 94 1/1 ceph-access cinder-ceph-key provider
keystone-credentials keystone keystone-k8s 211 1/1 identity-credentials keystone-credentials provider
keystone-endpoints keystone keystone-k8s 211 1/1 identity-service keystone provider
nova nova nova-k8s 106 1/1 nova-service nova provider
ovn-relay ovn-relay ovn-relay-k8s 95 1/1 ovsdb-cms-relay ovsdb-cms provider
rabbitmq rabbitmq rabbitmq-k8s 34 1/1 amqp rabbitmq provider
traefik-rgw traefik-rgw traefik-k8s 218 1/1 traefik-route traefik_route provider
```
## Vault status
To see status of Vault, run the following command:
```default
sunbeam vault status
```
Sample output of the above command looks like:
```default
Vault Status
+---------+-------------+-----------+
| Unit | Initialized | Sealed |
+=========+=============+===========+
| vault/0 | True | False |
| vault/1 | True | False |
| vault/2 | True | False |
+---------+-------------+-----------+
```
## Disabling Vault
To disable Vault, run the following command:
```default
sunbeam disable vault
```
#### CAUTION
Disabling Vault will completely remove it from the infrastructure,
all secrets will be lost.
# index.html.md
# LDAP Integration
This feature integrates the OpenStack
[Keystone](https://docs.openstack.org/keystone) service with an
external LDAP service. Effectively, the feature maps LDAP-based users to
cloud users via an OpenStack domain.
## Enabling LDAP
To enable the LDAP feature, run the following command:
```bash
sunbeam enable ldap
```
## Disabling LDAP
To disable the LDAP feature, run the following command:
```bash
sunbeam disable ldap
```
## Usage
### Adding a domain
Adding a domain refers to integrating Keystone with one or more existing
LDAP servers.
1. Create a YAML file with details of how Keystone should integrate with
the LDAP server. At a minimum, this should include a URL, user,
password, and suffix. See the [Keystone LDAP integration
guide](https://docs.openstack.org/keystone/2023.2/admin/configuration.html#integrate-identity-with-ldap)
for configuration guidance.
For example:
**dom1.yaml**:
```text
url: ldaps://ldap.example.com:636
user: cn=admin,dc=example,dc=com
password: mypassword
suffix: dc=example,dc=com
```
1. If the connection requires TLS, place the CA certificate in a file:
**dom1.cert**:
```text
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
1. Use the `sunbeam ldap add-domain` command to set up the domain,
adding the `--ca-cert-file` option if TLS is in use:
```bash
sunbeam ldap add-domain \
--domain-config-file ./dom1.yaml \
--ca-cert-file ./dom1.cert dom1
```
1. A new LDAP-backed domain will be created in Keystone. Verify this
with the native `openstack` CLI:
openstack user list –domain dom1
```text
+-------------------------------------------+---------------+
| | Name |
+-------------------------------------------+---------------+
| 941b5daa177ea518b5fc3b85fe9269729eb6abbb1 | John Hethel |
| d3b9d2bea306a049d4f56d30d6bba97b24c6db882 | Ryan Trunch |
| 7b699dc9a8037d6968c42c5b7b5d5a020d0f58e40 | Michael Diss |
+-------------------------------------------+---------------+
```
### Updating a domain
To update an LDAP domain the process is similar to adding one:
```bash
sunbeam ldap update-domain --domain-config-file ./dom1.yaml --ca-cert-file ./dom1.cert dom1
```
### Listing domains
To list LDAP domains:
```bash
sunbeam ldap list-domains
```
### Removing a domain
To remove an LDAP domain:
```bash
sunbeam ldap remove-domain ''
```
#### IMPORTANT
Since configuration (e.g. OpenStack projects) could have been made to the
domain after it was added, the `remove-domain` command only removes the
LDAP connection. To completely remove the domain, the `openstack` CLI
should be used (i.e. `openstack domain delete`).
# index.html.md
# Observability
This feature integrates Canonical OpenStack with (and optionally deploys) the
[Canonical Observability Stack
(COS)](https://charmhub.io/topics/canonical-observability-stack).
Sunbeam will automatically propagate default metrics and dashboards,
enabling you to effortlessly monitor the status of your single-node or
multi-node deployment of Sunbeam without the need for any additional
setup.
This feature provides ability to connect to an existing external COS or
deploy COS within the cloud.
## Connect to an existing COS
### Enabling Observability
[Register the external controller](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-external-juju-controllers.md)
hosting COS in Canonical OpenStack.
Ensure the Juju user has consume permissions granted on the
observability related offers.
To enable Sunbeam observability integration, run the following
command:
```bash
sunbeam enable observability external CONTROLLER GRAFANA_DASHBOARD_OFFER_URL PROMETHEUS_RECEIVE_REMOTE_WRITE_OFFER_URL LOKI_LOGGING_OFFER_URL
```
`CONTROLLER` is the name of external Juju controller that hosts COS.
`GRAFANA_DASHBOARD_OFFER_URL` is the remote Offer URL for Grafana
Dashboard.
`PROMETHEUS_RECEIVE_REMOTE_WRITE_OFFER_URL` is the remote Offer URL
for Prometheus.
`LOKI_LOGGING_OFFER_URL` is the remote Offer URL for Loki.
### Disabling Observability
To disable Observability, run the following command:
```bash
sunbeam disable observability external
```
## Deploy COS in Canonical OpenStack
-----END CERTIFICATE REQUEST-----
endpoint: amphora-controller-cert
relation_id: '215'
unit_name: null
- app_name: octavia
csr: |-
-----BEGIN CERTIFICATE REQUEST-----
-----END CERTIFICATE REQUEST-----
endpoint: amphora-issuing-ca
relation_id: '216'
unit_name: null
```
Extract each CSR and submit it to your CA to obtain the signed
certificates. The `endpoint` field identifies the purpose of each CSR:
| Endpoint | Certificate requirement |
|---------------------------|------------------------------------------------------|
| `amphora-controller-cert` | Leaf certificate (must **not** be a CA certificate). |
| `amphora-issuing-ca` | CA certificate (`basicConstraints: CA:TRUE`). |
#### Provide the signed certificates
Once your CA has returned the signed certificates, provide them to
Octavia:
```default
sunbeam loadbalancer provide_certificates
```
The command prompts for each certificate in turn. For both the
controller certificate and the issuing CA certificate, you will be asked
to supply:
- The signed certificate, base64-encoded (PEM).
- The CA certificate that signed it, base64-encoded (PEM).
- The full CA chain (intermediate + root CAs), base64-encoded (PEM) –
leave empty if the CA certificate is self-signed or no chain is
needed.
When all certificates have been accepted, the command confirms:
```default
TLS certificates provided to Octavia.
```
#### NOTE
If the Amphora feature is re-configured or certificates expire,
re-run `sunbeam loadbalancer list_outstanding_csrs` and
`sunbeam loadbalancer provide_certificates` to renew them.
## Disabling Load Balancer
To disable Load Balancer, run the following command:
```bash
sunbeam disable load-balancer
```
## Usage
Users should have roles `member` and `load-balancer_member` to
create and manage load balancers within their project.
Go through all the following sub-sections.
### Create a load balancer
Create a load balancer using the following command:
```default
openstack loadbalancer create --name --vip-network-id
```
Once the Amphora provider is enabled and configured (see
[Enabling the Amphora provider]()), it becomes the default provider.
To use the OVN provider instead, add `--provider ovn` to the
command.
For example, create the load balancer ‘test’:
```default
openstack loadbalancer create --name test --vip-network-id demo-network --wait
```
Sample output:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone | None |
| created_at | 2023-10-11T09:20:17 |
| description | |
| flavor_id | None |
| id | 8bb11dba-113e-46df-b7bd-3e099669dcf4 |
| listeners | |
| name | test |
| operating_status | ONLINE |
| pools | |
| project_id | cee090abc4d14819b9508e763e564984 |
| provider | ovn |
| provisioning_status | ACTIVE |
| updated_at | 2023-10-11T09:20:20 |
| vip_address | 192.168.122.218 |
| vip_network_id | 9cbb0646-9936-4ceb-9324-8f87ef118491 |
| vip_port_id | 749a598e-807c-475d-ab8d-26747bac2296 |
| vip_qos_policy_id | None |
| vip_subnet_id | 642d7a6d-625e-455a-a171-31082cd39c31 |
| tags | |
| additional_vips | [] |
+---------------------+--------------------------------------+
```
### Create a load balancer listener
Create a load balancer listener using the following command:
```default
openstack loadbalancer listener create --name --protocol --protocol-port
```
For example, add a listener on port 5555 for the ‘test’ load balancer:
```default
openstack loadbalancer listener create --name test-listener --protocol TCP --protocol-port 5555 test --wait
```
Sample output:
```default
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2023-10-11T09:21:11 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | 2412a8fa-ce0a-430b-80bb-5f8c8ec6168f |
| insert_headers | None |
| l7policies | |
| loadbalancers | 8bb11dba-113e-46df-b7bd-3e099669dcf4 |
| name | test-listener |
| operating_status | ONLINE |
| project_id | cee090abc4d14819b9508e763e564984 |
| protocol | TCP |
| protocol_port | 5555 |
| provisioning_status | ACTIVE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2023-10-11T09:21:12 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+
```
### Create a load balancer pool
Create a load balancer pool using the following command:
```default
openstack loadbalancer pool create --name --protocol --lb-algorithm --listener
```
For example, create the load balancer pool ‘test-pool’ for the
‘test-listener’ listener:
```default
openstack loadbalancer pool create --name test-pool --protocol TCP --lb-algorithm SOURCE_IP_PORT --listener test-listener --wait
```
Sample output:
```default
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2023-10-11T09:21:48 |
| description | |
| healthmonitor_id | |
| id | b7d9ac9f-5bfe-4786-a805-1a59fba98ee4 |
| lb_algorithm | SOURCE_IP_PORT |
| listeners | 2412a8fa-ce0a-430b-80bb-5f8c8ec6168f |
| loadbalancers | 8bb11dba-113e-46df-b7bd-3e099669dcf4 |
| members | |
| name | test-pool |
| operating_status | ONLINE |
| project_id | cee090abc4d14819b9508e763e564984 |
| protocol | TCP |
| provisioning_status | ACTIVE |
| session_persistence | None |
| updated_at | 2023-10-11T09:21:48 |
| tls_container_ref | None |
| ca_tls_container_ref | None |
| crl_container_ref | None |
| tls_enabled | False |
| tls_ciphers | None |
| tls_versions | None |
| tags | |
| alpn_protocols | None |
+----------------------+--------------------------------------+
```
### Add members to the load balancer pool
Add members to the load balancer pool using the following command:
```default
openstack loadbalancer member create --name --address --protocol-port
```
Run the above command multiple times to add new members to the load
balancer pool.
For example, to add member ‘test-pool-member1’ to the ‘test-pool’
pool, whose service is running on IP 192.168.122.183 and port 80:
```default
openstack loadbalancer member create --name test-pool-member1 --address 192.168.122.183 --protocol-port 80 test-pool --wait
```
Sample output:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| address | 192.168.122.183 |
| admin_state_up | True |
| created_at | 2023-10-11T09:23:23 |
| id | e386e580-8278-4253-8bbb-91f412d935e1 |
| name | test-pool-member1 |
| operating_status | NO_MONITOR |
| project_id | cee090abc4d14819b9508e763e564984 |
| protocol_port | 80 |
| provisioning_status | ACTIVE |
| subnet_id | None |
| updated_at | 2023-10-11T09:23:24 |
| weight | 1 |
| monitor_port | None |
| monitor_address | None |
| backup | False |
| tags | |
+---------------------+--------------------------------------+
```
### Add a health monitor to the load balancer pool
Add a health monitor to the load balancer pool using the following
command:
```default
openstack loadbalancer healthmonitor create --name --delay --timeout --max-retries --type
```
For example, to add health monitor ‘test-monitor’ to the ‘test-pool’
pool:
```default
openstack loadbalancer healthmonitor create --name test-monitor --delay 7 --timeout 5 --max-retries 3 --type TCP test-pool --wait
```
Sample output:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| project_id | cee090abc4d14819b9508e763e564984 |
| name | test-monitor |
| admin_state_up | True |
| pools | b7d9ac9f-5bfe-4786-a805-1a59fba98ee4 |
| created_at | 2023-10-11T09:33:33 |
| provisioning_status | ACTIVE |
| updated_at | 2023-10-11T09:33:34 |
| delay | 7 |
| expected_codes | None |
| max_retries | 3 |
| http_method | None |
| timeout | 5 |
| max_retries_down | 3 |
| url_path | None |
| type | TCP |
| id | 7f2cbe52-b024-4ede-a24b-7fa3cc6aa606 |
| operating_status | ONLINE |
| http_version | None |
| domain_name | None |
| tags | |
+---------------------+--------------------------------------+
```
Verify load balancer pool member operating status using the following
command:
```default
openstack loadbalancer member list
```
For example:
```default
openstack loadbalancer member list test-pool
```
Sample output:
```default
+--------------------------------------+-------------------+----------------------------------+---------------------+-----------------+---------------+------------------+--------+
| id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight |
+--------------------------------------+-------------------+----------------------------------+---------------------+-----------------+---------------+------------------+--------+
| e386e580-8278-4253-8bbb-91f412d935e1 | test-pool-member1 | cee090abc4d14819b9508e763e564984 | ACTIVE | 192.168.122.183 | 80 | ONLINE | 1 |
+--------------------------------------+-------------------+----------------------------------+---------------------+-----------------+---------------+------------------+--------+
```
### Verify the load balancer details
Verify the details of the load balancer using the following command:
```default
openstack loadbalancer status show
```
For example:
```default
openstack loadbalancer status show test
```
Sample output:
```default
{
"loadbalancer": {
"id": "8bb11dba-113e-46df-b7bd-3e099669dcf4",
"name": "test",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"listeners": [
{
"id": "2412a8fa-ce0a-430b-80bb-5f8c8ec6168f",
"name": "test-listener",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "b7d9ac9f-5bfe-4786-a805-1a59fba98ee4",
"name": "test-pool",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"health_monitor": {
"id": "7f2cbe52-b024-4ede-a24b-7fa3cc6aa606",
"name": "test-monitor",
"type": "TCP",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE"
},
"members": [
{
"id": "e386e580-8278-4253-8bbb-91f412d935e1",
"name": "test-pool-member1",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "192.168.122.183",
"protocol_port": 80
},
{
"id": "856fb894-714a-4d1d-beda-8cd2bc77485a",
"name": "test-pool-member2",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "192.168.122.248",
"protocol_port": 80
}
]
}
]
}
]
}
}
```
### Attach a floating IP address to the load balancer VIP port
To create a floating IP address and attach it to the load balancer VIP
port, use the below snippet:
```default
vip_port=$(openstack loadbalancer show test -c vip_port_id -f value)
fip_id=$(openstack floating ip create external-network -c ID -f value)
openstack floating ip set --port $vip_port $fip_id
lb_fip=$(openstack floating ip list --port $vip_port -c 'Floating IP Address' -f value)
echo $lb_fip
```
The above snippet outputs the load balancer VIP address:
```default
10.20.20.68
```
### Verify load balancer functionality
To verify load balancer functionality, apply the `nc` utility to the
load balancer VIP and listener port:
```default
nc -vz 10.20.20.68 5555
```
The output will report success if the load balancer connection to the
backend service is made:
```default
Connection to 10.20.20.68 5555 port [tcp/*] succeeded!
```
# index.html.md
# Resource Optimization
This feature deploys [Watcher](https://docs.openstack.org/watcher/latest/index.html), the
OpenStack Resource Optimization service.
## Enabling Resource Optimization
To enable Resource Optimization, run the following command:
```bash
sunbeam enable resource-optimization
```
## Disabling Resource Optimization
To disable Resource Optimization, run the following command:
```bash
sunbeam disable resource-optimization
```
## Usage
### List Goals
List goals using the following command:
```default
openstack optimize goal list
```
Sample output:
```default
+--------------------------------------+----------------------+----------------------+
| UUID | Name | Display name |
+--------------------------------------+----------------------+----------------------+
| 11ee813f-2ac3-4975-9529-706ba7025057 | airflow_optimization | Airflow Optimization |
| 38817441-5df3-4f9d-8fd7-58a966f2e921 | cluster_maintaining | Cluster Maintaining |
| 98504695-4052-43ec-a0a2-8cc945278fae | dummy | Dummy goal |
| 2413258c-bfba-4adb-aeda-b2ef611084a7 | hardware_maintenance | Hardware Maintenance |
| b92f38e9-7df7-4787-a7fd-29e5783e56f3 | noisy_neighbor | Noisy Neighbor |
| 48348b46-cca2-4669-be0c-336cea0e9396 | saving_energy | Saving Energy |
| f20ce098-df72-43ec-941e-ae72ad2ee0c6 | server_consolidation | Server Consolidation |
| 4c7b7c77-acc9-44f5-a082-dd728bdb9f0d | thermal_optimization | Thermal Optimization |
| 3c85d238-541d-479e-96e9-4038787c2fcf | unclassified | Unclassified |
| 7c1f150e-39b4-44e4-a352-ed947b71a9ae | workload_balancing | Workload Balancing |
+--------------------------------------+----------------------+----------------------+
```
### List Strategies in a goal
List the strategies for a goal using the following command:
```default
openstack optimize strategy list --goal GOAL
```
For example, list the strategies for goal ‘cluster_maintaining’:
```default
openstack optimize strategy list --goal cluster_maintaining
```
Sample output:
```default
+--------------------------------------+------------------+---------------------------+---------------------+
| UUID | Name | Display name | Goal |
+--------------------------------------+------------------+---------------------------+---------------------+
| 48351169-670f-4354-bac8-8d18061d1291 | host_maintenance | Host Maintenance Strategy | cluster_maintaining |
+--------------------------------------+------------------+---------------------------+---------------------+
```
### Create an Audit template
Create an Audit template using the following command:
```default
openstack optimize audittemplate create NAME GOAL --strategy STRATEGY
```
For example, create an audit template ‘host-maintenance-template’ for goal ‘cluster_maintaining’ and strategy ‘host_maintenance’:
```default
openstack optimize audittemplate create host-maintenance-template cluster_maintaining --strategy host_maintenance
```
Sample output:
```default
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| UUID | bb7caee4-f555-4d4f-89f4-7db627ce44cc |
| Created At | 2024-09-13T03:38:52.858848+00:00 |
| Updated At | None |
| Deleted At | None |
| Description | None |
| Name | host-maintenance-template |
| Goal | cluster_maintaining |
| Strategy | host_maintenance |
| Audit Scope | [] |
+-------------+--------------------------------------+
```
### Create an Audit
Create an Audit using the following command:
```default
openstack optimize audit create -a AUDIT_TEMPLATE_NAME -p key=value
```
For example, create an audit with template ‘host-maintenance-template’ and passing strategy parameters maintenance_node
```default
openstack optimize audit create -a host-maintenance-template -p maintenance_node=solqa-lab1-server-45.nosilo.lab1.solutionsqa
```
Sample output:
```default
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
| UUID | 2a4355b2-3e03-4a0f-80bf-92476e17b7da |
| Name | host_maintenance-2024-09-13T03:40:53.948011 |
| Created At | 2024-09-13T03:40:53.992685+00:00 |
| Updated At | None |
| Deleted At | None |
| State | PENDING |
| Audit Type | ONESHOT |
| Parameters | {'maintenance_node': 'solqa-lab1-server-45.nosilo.lab1.solutionsqa'} |
| Interval | None |
| Goal | cluster_maintaining |
| Strategy | host_maintenance |
| Audit Scope | [] |
| Auto Trigger | False |
| Next Run Time | None |
| Hostname | None |
| Start Time | None |
| End Time | None |
| Force | False |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
```
### Show the Audit details
Show the Audit details using the following command:
```default
openstack optimize audit show AUDIT_ID
```
Sample output:
```default
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
| UUID | 2a4355b2-3e03-4a0f-80bf-92476e17b7da |
| Name | host_maintenance-2024-09-13T03:40:53.948011 |
| Created At | 2024-09-13T03:40:54+00:00 |
| Updated At | 2024-09-13T03:41:07+00:00 |
| Deleted At | None |
| State | SUCCEEDED |
| Audit Type | ONESHOT |
| Parameters | {'maintenance_node': 'solqa-lab1-server-45.nosilo.lab1.solutionsqa'} |
| Interval | None |
| Goal | cluster_maintaining |
| Strategy | host_maintenance |
| Audit Scope | [] |
| Auto Trigger | False |
| Next Run Time | None |
| Hostname | watcher-0 |
| Start Time | None |
| End Time | None |
| Force | False |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------+
```
### List Action plan for an Audit
To list the Action plan for an Audit, run the following command:
```default
openstack optimize actionplan list --audit AUDIT_ID
```
Sample output:
```default
+--------------------------------------+--------------------------------------+-------------+------------+-----------------+
| UUID | Audit | State | Updated At | Global efficacy |
+--------------------------------------+--------------------------------------+-------------+------------+-----------------+
| 16e76af5-edbd-48b0-a443-946d921ca514 | 2a4355b2-3e03-4a0f-80bf-92476e17b7da | RECOMMENDED | None | |
+--------------------------------------+--------------------------------------+-------------+------------+-----------------+
```
### List actions
To list the actions in Action plan, run the following command:
```default
openstack optimize action list --action-plan ACTION_PLAN_ID
```
Sample output:
```default
+--------------------------------------+----------------------------------------------------------------------------------+---------+--------------------------------------+---------------------------+
| UUID | Parents | State | Action Plan | Action |
+--------------------------------------+----------------------------------------------------------------------------------+---------+--------------------------------------+---------------------------+
| d7f52ae0-37b5-456e-a9ac-a465bcce8aed | [] | PENDING | 16e76af5-edbd-48b0-a443-946d921ca514 | change_nova_service_state |
| 3f40421c-3f8d-4048-8b29-2c39bce9c16a | ['d7f52ae0-37b5-456e-a9ac-a465bcce8aed'] | PENDING | 16e76af5-edbd-48b0-a443-946d921ca514 | migrate |
| 617bcd97-e90b-4ba7-868c-548a96cd8408 | ['d7f52ae0-37b5-456e-a9ac-a465bcce8aed'] | PENDING | 16e76af5-edbd-48b0-a443-946d921ca514 | migrate |
| a5093f31-a9fd-41ef-a715-23761d282410 | ['3f40421c-3f8d-4048-8b29-2c39bce9c16a', '617bcd97-e90b-4ba7-868c-548a96cd8408'] | PENDING | 16e76af5-edbd-48b0-a443-946d921ca514 | migrate |
+--------------------------------------+----------------------------------------------------------------------------------+---------+--------------------------------------+---------------------------+
```
To list the detailed actions, run the following command:
```default
openstack optimize action list --action-plan ACTION_PLAN_ID --detail
```
### Start the Action plan
To start the action, run the following command:
```default
openstack optimize actionplan start ACTION_PLAN_ID
```
Sample output:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| UUID | 16e76af5-edbd-48b0-a443-946d921ca514 |
| Created At | 2024-09-13T03:41:06+00:00 |
| Updated At | 2024-09-13T03:45:32+00:00 |
| Deleted At | None |
| Audit | 2a4355b2-3e03-4a0f-80bf-92476e17b7da |
| Strategy | host_maintenance |
| State | PENDING |
| Efficacy indicators | [] |
| Global efficacy | [] |
| Hostname | None |
+---------------------+--------------------------------------+
```
### Show status of Action plan
To show the status of Action plan, run the following command:
```default
openstack optimize actionplan show ACTION_PLAN_ID
```
The state will be changed to SUCCEEDED once the actions complete.
Sample output:
```default
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| UUID | 16e76af5-edbd-48b0-a443-946d921ca514 |
| Created At | 2024-09-13T03:41:06+00:00 |
| Updated At | 2024-09-13T03:45:44+00:00 |
| Deleted At | None |
| Audit | 2a4355b2-3e03-4a0f-80bf-92476e17b7da |
| Strategy | host_maintenance |
| State | SUCCEEDED |
| Efficacy indicators | [] |
| | |
| Global efficacy | |
| Hostname | watcher-0 |
+---------------------+--------------------------------------+
```
## Limitations
1. Following goals are not supported:
> * airflow_optimization
> * thermal_optimization
> * noisy_neighbor
2. Strategies vm_workload_consolidation and workload_stabilization do not consider host memory usage in decision making as the metric hardware.memory.used is not currently collected.
# index.html.md
# Containers as a Service
This feature deploys [Magnum](https://docs.openstack.org/magnum/latest/index.html), the OpenStack CaaS service. Only
[Magnum CAPI Helm driver](https://docs.openstack.org/magnum-capi-helm/latest/) is supported.
This feature enables user to deploy [Canonical Kubernetes](https://ubuntu.com/kubernetes) on OpenStack infrastructure.
## Enabling CaaS
To enable CaaS, run the following command:
```bash
sunbeam enable caas
```
Use the OpenStack CLI to manage container infrastructures. See the
upstream [Magnum](https://docs.openstack.org/magnum/latest/index.html) documentation for details.
#### NOTE
The [Secrets as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/secrets.md) and [Load Balancer as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md) features are dependencies of the CaaS
feature. Make sure to enable them.
When using the CaaS feature in conjunction with the [Load Balancer as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md) feature, you
are subject to the same limitations as the latter feature. In particular, the OVN provider
only supports the `SOURCE_IP_PORT` load balancing algorithm.
## Disabling CaaS
To disable CaaS, run the following command:
```bash
sunbeam disable caas
```
## Usage
### Pre-requisites
1. Set the following properties on the image that will be used to deploy instances
* os-distro to ubuntu
* kube-version to workload cluster kubernetes version
```default
openstack image set IMAGE --os-distro ubuntu --property kube_version=v1.32.8
```
1. Ensure to add role member and load-balancer_member to the user as specified in
[Load Balancer as a Service](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/load-balancer.md)
### Create a kubernetes cluster
Create a cluster template using the following command:
```default
openstack coe cluster template \
create ck8s-cluster-template-ovn \
--image ubuntu \
--external-network external-network \
--flavor m1.medium \
--master-flavor m1.medium \
--master-lb-enabled \
--labels octavia_provider=ovn \
--labels octavia_lb_algorithm=SOURCE_IP_PORT \
--network-driver cilium \
--coe kubernetes
```
Sample output:
`ubuntu@sunbeam01:~$ ``openstack coe cluster template
> create ck8s-cluster-template-ovn \
> --image ubuntu \
> --external-network external-network \
> --flavor m1.medium \
> --master-flavor m1.medium \
> --master-lb-enabled \
> --labels octavia_provider=ovn \
> --labels octavia_lb_algorithm=SOURCE_IP_PORT \
> --network-driver cilium \
> --coe kubernetes`
```text
Request to create cluster template ck8s-cluster-template-ovn accepted
+-----------------------+-----------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------+
| insecure_registry | - |
| labels | {'octavia_provider': 'ovn', 'octavia_lb_algorithm': 'SOURCE_IP_PORT'} |
| updated_at | - |
| floating_ip_enabled | True |
| fixed_subnet | - |
| master_flavor_id | m1.medium |
| uuid | 1a3550d1-f493-449e-804f-7e1010cb1cf1 |
| no_proxy | - |
| https_proxy | - |
| tls_disabled | False |
| keypair_id | - |
| public | False |
| http_proxy | - |
| docker_volume_size | - |
| server_type | vm |
| external_network_id | external-network |
| cluster_distro | ubuntu |
| image_id | ubuntu |
| volume_driver | - |
| registry_enabled | False |
| docker_storage_driver | overlay2 |
| apiserver_port | - |
| name | ck8s-cluster-template-ovn |
| created_at | 2025-09-25T05:04:00.227248+00:00 |
| network_driver | cilium |
| fixed_network | - |
| coe | kubernetes |
| flavor_id | m1.medium |
| master_lb_enabled | True |
| dns_nameserver | 8.8.8.8 |
| project_id | 82c3eedc4a3646ef8777cf0d17a3ab32 |
| hidden | False |
| tags | - |
+-----------------------+-----------------------------------------------------------------------+
```
Create a Kubernetes cluster using the following command:
```text
openstack coe cluster create --cluster-template CLUSTER_TEMPLATE_UUID --master-count 3 --node-count 3 --timeout 900 sunbeam-ck8s-ovn
```
Sample output:
```text
Request to create cluster fc5724ae-aef8-4c89-aef8-78bc41f54325 accepted
```
Check cluster list status using the following command:
```text
openstack coe cluster list
+--------------------------------------+------------------+---------+------------+--------------+-----------------+---------------+
| uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+------------------+---------+------------+--------------+-----------------+---------------+
| fc5724ae-aef8-4c89-aef8-78bc41f54325 | sunbeam-ck8s-ovn | None | 3 | 3 | CREATE_COMPLETE | HEALTHY |
+--------------------------------------+------------------+---------+------------+--------------+-----------------+---------------+
```
#### NOTE
You may need to wait a few minutes before the cluster is ready.
Check cluster status using the following command:
```text
openstack coe cluster show CLUSTER_UUID
+----------------------+------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+------------------------------------------------------------------------------------------------+
| status | CREATE_COMPLETE |
| health_status | HEALTHY |
| cluster_template_id | 1a3550d1-f493-449e-804f-7e1010cb1cf1 |
| node_addresses | [] |
| uuid | fc5724ae-aef8-4c89-aef8-78bc41f54325 |
| stack_id | sunbeam-ck8s-ovn-sei7c6sxbikj |
| status_reason | None |
| created_at | 2025-09-25T05:18:52+00:00 |
| updated_at | 2025-09-25T05:28:52+00:00 |
| coe_version | v1.32.8 |
| labels | {'octavia_provider': 'ovn', 'octavia_lb_algorithm': 'SOURCE_IP_PORT'} |
| labels_overridden | {} |
| labels_skipped | {} |
| labels_added | {} |
| fixed_network | None |
| fixed_subnet | None |
| floating_ip_enabled | True |
| faults | |
| keypair | None |
| api_address | https://172.16.2.247:6443 |
| master_addresses | [] |
| master_lb_enabled | True |
| create_timeout | 900 |
| node_count | 3 |
| discovery_url | None |
| docker_volume_size | None |
| master_count | 3 |
| container_version | None |
| name | sunbeam-ck8s-ovn |
| master_flavor_id | m1.medium |
| flavor_id | m1.medium |
| health_status_reason | {'cluster': 'Ready', 'infrastructure': 'Ready', 'controlplane': 'Ready', 'nodegroup': 'Ready'} |
| project_id | 82c3eedc4a3646ef8777cf0d17a3ab32 |
+----------------------+------------------------------------------------------------------------------------------------+
```
Access your Kubernetes cluster using the following commands:
```text
mkdir config-dir
openstack coe cluster config sunbeam-k8s-ovn --dir config-dir/
export KUBECONFIG=/home/ubuntu/config-dir/config
sudo -E k8s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-7c98r 1/1 Running 0 21m
kube-system cilium-lk2w9 1/1 Running 0 21m
kube-system cilium-operator-6fb79c547b-h8ds7 1/1 Running 0 24m
kube-system cilium-p5wz7 1/1 Running 0 24m
kube-system cilium-pmcj8 1/1 Running 0 19m
kube-system cilium-tz5sj 1/1 Running 0 21m
kube-system cilium-vs6m5 1/1 Running 0 21m
kube-system ck-storage-rawfile-csi-controller-0 2/2 Running 0 25m
kube-system ck-storage-rawfile-csi-node-6bn6l 4/4 Running 0 21m
kube-system ck-storage-rawfile-csi-node-7gndg 4/4 Running 0 21m
kube-system ck-storage-rawfile-csi-node-cjgtk 4/4 Running 0 21m
kube-system ck-storage-rawfile-csi-node-fl8fs 4/4 Running 0 25m
kube-system ck-storage-rawfile-csi-node-hc4pj 4/4 Running 0 19m
kube-system ck-storage-rawfile-csi-node-zrn5z 4/4 Running 0 21m
kube-system coredns-fc9c778db-fzrdf 1/1 Running 0 25m
kube-system k8sd-proxy-g7xvc 1/1 Running 0 20m
kube-system k8sd-proxy-jqxp5 1/1 Running 0 20m
kube-system k8sd-proxy-k7bv2 1/1 Running 0 18m
kube-system k8sd-proxy-qwzr2 1/1 Running 0 20m
kube-system k8sd-proxy-v2jqt 1/1 Running 0 23m
kube-system k8sd-proxy-vv6t2 1/1 Running 0 21m
kube-system metrics-server-8694c96fb7-hkfk6 1/1 Running 0 25m
kubernetes-dashboard kubernetes-dashboard-1758777722-api-574545d7f4-69bcx 1/1 Running 0 24m
kubernetes-dashboard kubernetes-dashboard-1758777722-auth-7b949ccdd9-d54tv 1/1 Running 0 24m
kubernetes-dashboard kubernetes-dashboard-1758777722-kong-58bc8dc74b-gl5n2 1/1 Running 0 24m
kubernetes-dashboard kubernetes-dashboard-1758777722-metrics-scraper-75cd94bbc-2tnq6 1/1 Running 0 24m
kubernetes-dashboard kubernetes-dashboard-1758777722-web-5866567c7d-cp6dg 1/1 Running 0 24m
metallb-system metallb-controller-7f647445fc-5ztvp 1/1 Running 0 25m
metallb-system metallb-speaker-8jdfv 1/1 Running 0 20m
metallb-system metallb-speaker-bqwv2 1/1 Running 0 18m
metallb-system metallb-speaker-gx5j6 1/1 Running 0 21m
metallb-system metallb-speaker-s2gg9 1/1 Running 0 20m
metallb-system metallb-speaker-vjfrf 1/1 Running 0 20m
metallb-system metallb-speaker-zfv27 1/1 Running 0 23m
openstack-system openstack-cinder-csi-controllerplugin-5944b6858f-6wx82 6/6 Running 0 24m
openstack-system openstack-cinder-csi-nodeplugin-2gsb5 3/3 Running 0 21m
openstack-system openstack-cinder-csi-nodeplugin-2m4gb 3/3 Running 0 19m
openstack-system openstack-cinder-csi-nodeplugin-bzvcv 3/3 Running 0 21m
openstack-system openstack-cinder-csi-nodeplugin-cbl9r 3/3 Running 0 24m
openstack-system openstack-cinder-csi-nodeplugin-jnx2z 3/3 Running 0 21m
openstack-system openstack-cinder-csi-nodeplugin-wwhx6 3/3 Running 0 21m
openstack-system openstack-cloud-controller-manager-6w8v2 1/1 Running 0 20m
openstack-system openstack-cloud-controller-manager-sh2zr 1/1 Running 0 24m
openstack-system openstack-cloud-controller-manager-vbnlt 1/1 Running 0 18m
```
Currently the command openstack coe cluster config is not returning proper kubeconfig.
As a workaround, get the kubeconfig using clusterctl
```text
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.4/clusterctl-linux-amd64 -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl
sudo k8s config > kubeconfig
KUBECONFIG=kubeconfig clusterctl get kubeconfig --namespace magnum- > config-dir/config
```
Replace PROJECT_ID and CLUSTER_STACK_ID from the output values in openstack coe cluster show.
### Enable Autoscaling
To enable Autoscaling feature, the cluster template should have the following labels
```text
--labels auto_scaling_enabled=True
--labels min_node_count=3
--labels max_node_count=5
```
#### NOTE
The value for –node-count in openstack coe cluster create command should be in the range [min_node_count … max_node_count]
### Enable Keystone authentication and authorization webhook for Workload Kubernetes Cluster
To enable [Keystone authentication and authorization feature](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/keystone-auth/using-keystone-webhook-authenticator-and-authorizer.md), the cluster template should have the following label
```text
--labels keystone_auth_enabled=True
```
Existing cluster can be upgraded to enable keystone auth by running the following command
`ubuntu@sunbeam01:~$ ``openstack coe cluster upgrade CLUSTER_UUID NEW_CLUSTER_TEMPLATE_UUID`
```text
Request to upgrade cluster fc5724ae-aef8-4c89-aef8-78bc41f54325 has been accepted.
```
#### NOTE
You may need to wait a few minutes before the cluster is ready.
To verify if keystone-auth is enabled or not, run the following command
```text
sudo -E k8s kubectl -n kube-system get po -l app.kubernetes.io/name=k8s-keystone-auth
NAME READY STATUS RESTARTS AGE
k8s-keystone-auth-1758780472-7qkbc 1/1 Running 0 7m56s
k8s-keystone-auth-1758780472-kwlzj 1/1 Running 0 2m46s
k8s-keystone-auth-1758780472-wprwk 1/1 Running 0 6m21s
```
The default keystone-k8s auth policy is specified [here](https://github.com/catalyst-cloud/capi-plugin-helm-charts/blob/main/charts/k8s-keystone-auth/values.yaml#L80).
For custom policies, User need to manually update the config map k8s-keystone-auth-policy
in kube-system namespace and recycle the k8s-keystone-auth pods.
### Setup a Kubernetes cluster in a proxy environment
To setup a kubernetes cluster in a proxy environment, set the following parameters
in openstack coe cluster template create command
```text
--http-proxy <>
--https-proxy <>
--no-proxy <>
```
### Delete a Cluster
Delete the kubernetes cluster using the following command:
```text
openstack coe cluster delete CLUSTER_UUID
```
#### NOTE
Cluster deletion fails as ingress is enabled in Canonical Kubernetes CAPI
deployment but ingress is not supported in Magnum CAPI Helm driver.
As a workaround, delete the loadbalancer using the following command:
openstack loadbalancer delete kube_service__kube-system_cilium-ingress –cascade
## Limitations:
* Only Cilium network driver is supported.
* Enabling monitoring feature via label monitoring_enabled is not supported.
* Enabling Registry mirrors is not supported in Magnum CAPI Helm driver.
# index.html.md
# DNS as a Service
This feature deploys [Designate](https://docs.openstack.org/designate/latest/index.html), the OpenStack DNS service.
## Enabling DNS
To enable DNS, run the following command:
```default
sunbeam enable dns ""
```
The openstack CLI can now be used to manage DNS. See the upstream
[Designate command-line interface documentation](https://docs.openstack.org/python-designateclient/latest/user/shell-v2.html) for details.
Nameservers are specified with FQDNs separated by a space, each ending
with a dot, whose records point to the DNS instance managed by the
Designate service. It is assumed that your infrastructure DNS is
configured to redirect your nameserver records to the DNS service
address.
## Disabling DNS
To disable DNS, run the following command:
```default
sunbeam disable dns
```
## Fetching DNS service address
To fetch the DNS service address, run the following command:
```default
sunbeam dns address
```
## Usage
Users need the role `member` to be able to manage DNS zones and
records. A user has this role by default so all users have the ability
to manage DNS in their own project.
For example, create zone `sunbeam.tld` with:
```default
openstack zone create --email dnsmaster@sunbeam.tld sunbeam.tld.
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | CREATE |
| attributes | |
| created_at | 2023-10-11T20:25:52.000000 |
| description | None |
| email | dnsmaster@sunbeam.tld |
| id | f27cd25d-43ff-4205-84a4-79c524bd9652 |
| masters | |
| name | sunbeam.tld. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | b6cc0f4bf25c432785b4f7c91858304b |
| serial | 1697055952 |
| shared | False |
| status | PENDING |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | None |
| version | 1 |
+----------------+--------------------------------------+
```
Retrieve the list of DNS zones - wait for the new zone to become
`ACTIVE`:
```default
openstack zone list
+--------------------------------------+--------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+--------------------------------------+--------------+---------+------------+--------+--------+
| f27cd25d-43ff-4205-84a4-79c524bd9652 | sunbeam.tld. | PRIMARY | 1697055952 | ACTIVE | NONE |
+--------------------------------------+--------------+---------+------------+--------+--------+
```
Create the `TXT` record `note.sunbeam.tld`:
```default
openstack recordset create --type TXT --record '"This is a record created in Sunbeam!"' sunbeam.tld. note
+-------------+----------------------------------------+
| Field | Value |
+-------------+----------------------------------------+
| action | CREATE |
| created_at | 2023-10-11T20:30:33.000000 |
| description | None |
| id | 40222abd-1624-42af-90ff-7fc212e99885 |
| name | note.sunbeam.tld. |
| project_id | b6cc0f4bf25c432785b4f7c91858304b |
| records | "This is a record created in Sunbeam!" |
| status | PENDING |
| ttl | None |
| type | TXT |
| updated_at | None |
| version | 1 |
| zone_id | f27cd25d-43ff-4205-84a4-79c524bd9652 |
| zone_name | sunbeam.tld. |
+-------------+----------------------------------------+
```
Obtain the address of the DNS service with the `sunbeam` command:
```default
sunbeam dns address
10.206.54.244
```
With the `dig` command, query the DNS service and verify that it
returns the newly-created `TXT` record:
```default
dig @10.206.54.244 +short TXT note.sunbeam.tld
"This is a record created in Sunbeam!"
```
# index.html.md
# Validation
## Overview
This feature deploys [Tempest](https://charmhub.io/tempest-k8s), a
tool for running integration tests, which includes tests for OpenStack
API validation, scenarios, and other specific tests useful in validating
an OpenStack deployment.
## Enable Validation
To enable Validation, run the following command:
```bash
sunbeam enable validation
```
#### NOTE
During the initialization of Tempest, multiple network resources—including routers, networks, subnets, and ports—will be created on OpenStack.
Currently, Tempest requires 40 external IP addresses for router gateways.
Please ensure that at least 40 external IP addresses are available before enabling validation.
## Disable Validation
To disable Validation, run the following command:
```bash
sunbeam disable validation
```
## Usage
Validation tests can be run on-demand and they can be scheduled to run
in the background.
### On-demand validation
To run a one-time validation test against the deployment, run the
following command:
```default
sunbeam validation run [profile]
```
If no profile is specified, the validation test defaults to running the
appropriate RefStack profile for the OpenStack version deployed. You can
check the list of available profiles provided by running:
```default
sunbeam validation profiles
```
Sample output:
```default
Available profiles
Name Description
────────────────────────────────────────────────────────────────────────────────────────
refstack Tests that are part of the RefStack project https://refstack.openstack.org/
quick A short list of tests for quick validation
smoke Tests tagged as "smoke"
all All tests (very large number, not usually recommended)
```
A summary of the validation result will be printed out to the screen
upon completion of the command:
```default
Totals
======
Ran: 115 tests in 1036.6368 sec.
- Passed: 70
- Skipped: 19
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 26
Sum of execute time for each test: 718.3638 sec.
```
In order to obtain detailed test results, use the `--output` option
and specify a local path to save the result file to:
```default
sunbeam validation run --output ./validation.log
```
The result of the most recent run can be retrieved with:
```default
sunbeam validation get-last-result --output ./validation.log
```
### Periodic validation
If the [Observability](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md) feature is also enabled,
periodic validation tests will be performed using the `quick` profile.
By default, periodic checks are scheduled to execute on an hourly basis.
You can configure this frequency to a 5-field
[cron](https://en.wikipedia.org/wiki/Cron) schedule, which specifies
the interval between checks. For example, if you would like periodic
checks to be run every 6 hours on the first minute, run the following
command:
```default
sunbeam configure validation schedule="1 */6 * * *"
```
Periodic checks can also be disabled by setting the schedule parameter
to an empty string:
```default
sunbeam configure validation schedule=""
```
#### NOTE
Due to performance considerations, intervals under 15 minutes are not supported.
Results will be displayed in a validation-specific Grafana dashboard and
alerts will be fired when periodic checks fail. For more information on
how to access Grafana dashboards and receive alerts, please refer to the
[Observability](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/observability.md) feature documentation.
# index.html.md
# Orchestration
This feature deploys [Heat](https://docs.openstack.org/heat), the
OpenStack Orchestration service.
## Enabling Orchestration
To enable Orchestration, run the following command:
```bash
sunbeam enable orchestration
```
Use OpenStack CLI to manage orchestration stacks. See the upstream [Heat
documentation](https://docs.openstack.org/heat/latest/getting_started/create_a_stack.html)
for details.
## Disabling Orchestration
To disable Orchestration, run the following command:
```default
sunbeam disable orchestration
```
This will terminate the application but not remove it from the model. To
do that, run the following:
```default
juju remove-application --force --no-wait --no-prompt -m openstack \
heat heat-cfn heat-mysql-router heat-cfn-mysql-router
```
## Usage
Create a Heat stack using the following command:
```default
openstack stack create \
-t https://opendev.org/openstack/heat-templates/raw/branch/master/hot/servers_in_existing_neutron_net.yaml \
--parameter key_name=sunbeam \
--parameter image=ubuntu \
--parameter flavor=m1.tiny \
--parameter public_net_id=external-network \
--parameter private_net_id=demo-network \
--parameter private_subnet_id=demo-subnet \
teststack
```
The Heat template referred to in the above command creates two servers
in network `demo-subnet` and assigns them floating IP addresses.
Sample output:
```default
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 952770ed-f40e-4547-8f22-4dba8b1c714b |
| stack_name | teststack |
| description | HOT template to deploy two servers into an existing neutron tenant network and assign floating IP addresses to each server so they are routable from the public network. |
| | |
| creation_time | 2023-10-13T06:47:35Z |
| updated_time | None |
| stack_status | CREATE_IN_PROGRESS |
| stack_status_reason | Stack CREATE started |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
Verify stack status and wait for completion:
```default
openstack stack list
```
Sample output:
```default
+--------------------------------------+------------+-----------------+----------------------+--------------+
| ID | Stack Name | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+-----------------+----------------------+--------------+
| 952770ed-f40e-4547-8f22-4dba8b1c714b | teststack | CREATE_COMPLETE | 2023-10-13T06:47:35Z | None |
+--------------------------------------+------------+-----------------+----------------------+--------------+
```
Get stack details using the below command:
```default
openstack stack show teststack
```
Sample output:
```default
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 952770ed-f40e-4547-8f22-4dba8b1c714b |
| stack_name | teststack |
| description | HOT template to deploy two servers into an existing neutron tenant network and assign floating IP addresses to each server so they are routable from the public network. |
| | |
| creation_time | 2023-10-13T06:47:35Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
| parameters | OS::project_id: 098db89856cb4306828b0be678667294 |
| | OS::stack_id: 952770ed-f40e-4547-8f22-4dba8b1c714b |
| | OS::stack_name: teststack |
| | flavor: m1.tiny |
| | image: ubuntu |
| | key_name: sunbeam |
| | private_net_id: demo-network |
| | private_subnet_id: demo-subnet |
| | public_net_id: external-network |
| | |
| outputs | - description: IP address of server1 in private network |
| | output_key: server1_private_ip |
| | output_value: 192.168.122.154 |
| | - description: Floating IP address of server2 in public network |
| | output_key: server2_public_ip |
| | output_value: 10.20.20.157 |
| | - description: Floating IP address of server1 in public network |
| | output_key: server1_public_ip |
| | output_value: 10.20.20.73 |
| | - description: IP address of server2 in private network |
| | output_key: server2_private_ip |
| | output_value: 192.168.122.157 |
| | |
| links | - href: http://10.20.21.13/openstack-heat/v1/098db89856cb4306828b0be678667294/stacks/teststack/952770ed-f40e-4547-8f22-4dba8b1c714b |
| | rel: self |
| | |
| deletion_time | None |
| notification_topics | [] |
| capabilities | [] |
| disable_rollback | True |
| timeout_mins | None |
| stack_owner | demo |
| parent | None |
| stack_user_project_id | 0dfde376b8544b0499e74c4d7a82cc27 |
| tags | [] |
| | |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
Verify if stack resources are created (i.e. if new servers are launched
or not):
```default
openstack server list
+--------------------------------------+----------+--------+--------------------------------------------+--------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------+--------+--------------------------------------------+--------+---------+
| 0ad5e745-8d5b-4cc3-8ccf-f460733a3af4 | Server2 | ACTIVE | demo-network=10.20.20.157, 192.168.122.157 | ubuntu | m1.tiny |
| 07261def-a40b-4976-9399-0398319b4067 | Server1 | ACTIVE | demo-network=10.20.20.73, 192.168.122.154 | ubuntu | m1.tiny |
+--------------------------------------+----------+--------+--------------------------------------------+--------+---------+
```
# index.html.md
# Images Sync
This feature deploys OpenStack Images Sync, a tool for importing
images from a SimpleStreams source to the OpenStack Glance image service.
## Enable Images Sync
To enable Images Sync, run the following command:
```bash
sunbeam enable images-sync
```
## Disable Images Sync
To disable Images Sync, run the following command:
```bash
sunbeam disable images-sync
```
#### CAUTION
**Caution**: Disabling Images Sync will **not** remove images that have been
previously imported from the Glance image service.
## Usage
Users need the role `reader` to list images.
To list images added by the Images Sync feature, run the following
command:
```default
openstack image list | grep auto-sync/
```
Sample output:
```default
| 200df230-0983-4cd8-9d14-97327664f77b | auto-sync/ubuntu-focal-20.04-amd64-server-20240513-disk1.img | active |
| 1935961b-e646-4f0d-a796-8c653308f790 | auto-sync/ubuntu-jammy-22.04-amd64-server-20240514-disk1.img | active |
| 62be8807-f068-4317-9552-c1357fa8d962 | auto-sync/ubuntu-noble-24.04-amd64-server-20240523.1-disk1.img | active |
```
The feature downloads images for the three most recent LTS releases.
# index.html.md
# Shared Filesystems as a Service
This feature deploys [Manila](https://docs.openstack.org/manila/latest/index.html), the OpenStack Shared Filesystems service.
It enables the user to create NFS Shared Filesystems on the Ceph storage
backend.
## Enabling Shared Filesystems
This feature requires the storage role. To enable this feature, run the
following command:
```default
sunbeam enable shared-filesystem
```
The openstack CLI can now be used to create and manage CephFS NFS Shared
Filesystems. See the upstream [Manila CLI](https://docs.openstack.org/python-manilaclient/latest/cli/osc_plugin_cli.html) documentation for details.
## Disabling Shared Filesystems
To disable this feature, run the following command:
```default
sunbeam disable shared-filesystem
```
## Usage
Administrators need to create a share type suitable for CephFS NFS share:
```default
openstack share type create cephfsnfstype false
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| id | 8fa7d7ff-16de-44c4-97c2-627b608970bd |
| name | cephfsnfstype |
| visibility | public |
| is_default | False |
| required_extra_specs | driver_handles_share_servers : False |
| optional_extra_specs | |
| description | None |
+----------------------+--------------------------------------+
openstack share type set cephfsnfstype --extra-specs vendor_name=Ceph storage_protocol=NFS
```
Shares can then be created with the following command:
```default
openstack share create --share-type cephfsnfstype --name cephnfsshare1 nfs 1
+---------------------------------------+--------------------------------------+
| Field | Value |
+---------------------------------------+--------------------------------------+
| access_rules_status | active |
| availability_zone | None |
| create_share_from_snapshot_support | False |
| created_at | 2025-08-18T07:19:04.825882 |
| description | None |
| has_replicas | False |
| host | |
| id | 9ad6e4f1-26c2-47b0-944a-957c973e8260 |
| is_public | False |
| is_soft_deleted | False |
| metadata | {} |
| mount_snapshot_support | False |
| name | cephnfsshare2 |
| progress | None |
| project_id | 59e4d08bafeb42a2987d0dd7ef477764 |
| replication_type | None |
| revert_to_snapshot_support | False |
| scheduled_to_be_deleted_at | None |
| share_group_id | None |
| share_network_id | None |
| share_proto | NFS |
| share_server_id | None |
| share_type | 89a7c4c9-9f01-4051-a1f2-407c63387e68 |
| share_type_name | cephfsnfstype |
| size | 1 |
| snapshot_id | None |
| snapshot_support | False |
| source_backup_id | None |
| source_share_group_snapshot_member_id | None |
| status | creating |
| task_state | None |
| user_id | 03ed09d378be484cade24f5731bb820d |
| volume_type | cephfsnfstype |
+---------------------------------------+--------------------------------------+
```
The created share should be available:
```default
openstack share list
+--------------------------------------+---------------+------+-------------+-----------+-----------+-----------------+--------------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+---------------+------+-------------+-----------+-----------+-----------------+--------------------------------+-------------------+
| 5c156c74-43bf-432b-af2e-bddc82f5c6f9 | cephnfsshare1 | 1 | NFS | available | False | cephfsnfstype | manila-cephfs-0@cephnfs#cephfs | nova |
+--------------------------------------+---------------+------+-------------+-----------+-----------+-----------------+--------------------------------+-------------------+
```
Note the export location of the share:
```default
openstack share export location list cephnfsshare1
+--------------------------------------+------------------------------------------------------------------------------------------------------------+-----------+
| ID | Path | Preferred |
+--------------------------------------+------------------------------------------------------------------------------------------------------------+-----------+
| 96f2ae5a-7fdf-4457-a65d-6b8579635dd0 | 192.168.137.10:/volumes/_nogroup/5692f246-d3d0-4567-81a6-3f590d1957a4/aa0c7383-b5e1-48ab-984c-15b6219c48e7 | True |
+--------------------------------------+------------------------------------------------------------------------------------------------------------+-----------+
```
The export location of the share contains the bind IP address of the NFS
Ganesha server and the path to be mounted.
# index.html.md
# Secrets as a Service
This feature deploys [Barbican](https://docs.openstack.org/barbican/latest/index.html), the OpenStack Key Manager service.
## Enabling Secrets
To enable Secrets, run the following command:
```bash
sunbeam enable secrets
```
The openstack CLI can now be used to manage Secrets. See the upstream
[Barbican CLI](https://docs.openstack.org/python-barbicanclient/latest/cli/) documentation for details.
#### NOTE
The [Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md) feature is a dependency of the
Secrets feature and should be enabled prior to enabling Barbican.
## Disabling Secrets
To disable Secrets, run the following command:
```bash
sunbeam disable secrets
```
## Usage
Users need the role `creator` to be able to create / read / destroy
secrets.
Verify if a user belongs to this role with (admin rights needed):
```default
openstack role assignment list --user --role creator
+----------------------------------+----------------------------------+-------+----------------------------------+--------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+--------+-----------+
| 3ef18094c76a403291ccf727851616ae | 4f2e8ef6b897403fb9865123b7b57a34 | | 3e5bb39a247b471494e051ae8d0530fb | | | False |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+--------+-----------+
```
Create a secret consisting of the string `my_payload`, and request
just the `Secret href` field as output:
```default
openstack secret store --name my_secret --payload my_payload -c "Secret href"
+-------------+-----------------------------------------------------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------------------------------------------------+
| Secret href | http://10.206.54.241/openstack-barbican/v1/secrets/65ad38a3-811e-4445-8472-13aa2fa5042d |
+-------------+-----------------------------------------------------------------------------------------+
```
Retrieve the original secret (`my_payload`) via the secret href value:
```default
openstack secret get --payload http://10.206.54.241/openstack-barbican/v1/secrets/65ad38a3-811e-4445-8472-13aa2fa5042d
+---------+-------------+
| Field | Value |
+---------+-------------+
| Payload | my_payload |
+---------+-------------+
```
\
--user-domain service_domain \
--project services \
--project-domain service_domain \
secret-decrypter
```
# index.html.md
# Object Storage
The object storage service providing Swift and S3 endpoints is enabled
automatically as soon as there are storage nodes in a cluster.
The object storage endpoints can be retrieved using `openstack`
client. See [Using OpenStack CLI](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/using-the-openstack-cli.md) on how to use OpenStack
CLI.
Run the below command to get s3 endpoint:
```default
openstack endpoint list --service s3 --interface public
```
Sample output of the above command:
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------+
| 74c9adaa7041422692a1f55adf0a65eb | RegionOne | s3 | s3 | True | public | http://10.20.21.10 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------+
```
Run the below command to get swift endpoint:
```default
openstack endpoint list --service swift --interface public
```
Sample output of the above command:
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------+
| d30b885de04b4b1090d2c7d05d4c6562 | RegionOne | swift | object-store | True | public | http://10.20.21.10/swift/v1/AUTH_$(project_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------+
```
## Usage
To access using the swift protocol, users can use the `openstack`
client.
Create a Swift container using command:
```default
openstack container create foo
```
Sample output:
```default
+---------------------------------------+-----------+--------------------------------------------------+
| account | container | x-trans-id |
+---------------------------------------+-----------+--------------------------------------------------+
| AUTH_75a27eb3202d4fcda251647d1d78af2c | foo | tx0000045042a8694f86bc4-0066877b39-121b1-default |
+---------------------------------------+-----------+--------------------------------------------------+
```
List containers using command:
```default
openstack container list
```
Sample output:
```default
+------+
| Name |
+------+
| foo |
+------+
```
Upload an object in the container using command:
```default
openstack object create foo test.txt --name test
```
Sample output:
```default
+--------+-----------+----------------------------------+
| object | container | etag |
+--------+-----------+----------------------------------+
| test | foo | d8e8fca2dc0f896fd7cb4cb0031ba249 |
+--------+-----------+----------------------------------+
```
List the objects in the container using command:
```default
openstack object list foo
```
Sample output:
```default
+------+
| Name |
+------+
| test |
+------+
```
Delete an object from container using command:
```default
openstack object delete foo test
```
Delete a container using command:
```default
openstack container delete foo
```
# index.html.md
# Ubuntu Pro
## Overview
This feature enables [Ubuntu Pro](https://ubuntu.com/pro) support for
Canonical OpenStack. Ubuntu Pro provides additional benefits such as Livepatch
and extended security support periods for Ubuntu LTS.
Ubuntu Pro is free for [limited personal
usage](https://ubuntu.com/pro/dashboard) or subscriptions can be
[purchased for commercial support](https://ubuntu.com/pro/subscribe).
## Enabling Ubuntu Pro
To enable Ubuntu Pro support, run the following command with an
attachment token associated with your subscription:
```default
sunbeam enable pro
```
## Disabling Ubuntu Pro
To disable Ubuntu Pro support, run the following command:
```default
sunbeam disable pro
```
## Usage
To check the Ubuntu Pro status on any node in the Canonical OpenStack
deployment login to the node and use the `pro` command to validate the
subscription attachment and enabled services:
```default
pro status
```
Example output is:
```default
SERVICE ENTITLED STATUS DESCRIPTION
esm-apps yes enabled Expanded Security Maintenance for Applications
esm-infra yes enabled Expanded Security Maintenance for Infrastructure
livepatch yes disabled Canonical Livepatch service
realtime-kernel* yes disabled Ubuntu kernel with PREEMPT_RT patches integrated
usg yes disabled Security compliance and audit tools
* Service has variants
For a list of all Ubuntu Pro services and variants, run 'pro status --all'
Enable services with: pro enable
Account: microstack@ubuntu.com
Subscription: Ubuntu Pro - free personal subscription
```
# index.html.md
# Baremetal as a Service
This feature deploys [Ironic](https://docs.openstack.org/ironic/latest/index.html), the bare metal provisioning service for
OpenStack. It allows OpenStack users to provision bare metal machines,
as opposed to virtual machines.
## Enabling Baremetal
This feature requires the storage role. To enable this feature, run the
following command:
```default
sunbeam enable baremetal
```
The openstack CLI can now be used to manage bare metal machines. See the
upstream [Ironic CLI](https://docs.openstack.org/python-ironicclient/latest/index.html) documentation for details.
The feature will be configured based on the cluster’s manifest file.
Alternatively, a different manifest file can be specified during the feature
enablement:
```default
sunbeam enable --manifest baremetal-manifest.yaml baremetal
```
Sample baremetal-manifest.yaml file:
```yaml
features:
baremetal:
software:
charms:
ironic-conductor-k8s:
channel: 2025.1/edge
ironic-k8s:
channel: 2025.1/edge
nova-ironic-k8s:
channel: 2025.1/edge
neutron-baremetal-switch-config-k8s:
channel: 2025.1/edge
neutron-generic-switch-config-k8s:
channel: 2025.1/edge
config:
shards: ["shard0", "shard1"]
conductor-groups: ["shard0", "shard1"]
switchconfigs:
netconf:
nexus:
configfile: |
[nexus.example.net]
driver = netconf-openconfig
device_params = name:nexus
switch_info = nexus
switch_id = 00:53:00:0a:0a:0a
host = nexus.example.net
username = user
key_filename = /etc/neutron/sshkeys/nexus-sshkey
additional-files:
nexus-sshkey: |
some key here.
generic:
arista:
configfile: |
[genericswitch:arista-hostname]
device_type = netmiko_arista_eos
ngs_mac_address = 00:53:00:0a:0a:0a
ip = 10.20.30.40
username = admin
key_file = /etc/neutron/sshkeys/arista-key
additional-files:
arista-key: |
some key here.
```
#### NOTE
Rerunning the sunbeam enable baremetal command with a different manifest
file will replace the previously deployed feature configuration (e.g.:
deployed nova-ironic shards, Ironic Conductor groups, Neutron switch
configurations).
For the switch configurations, the following restrictions apply:
- The key_filename and key_file config options base file paths must be
/etc/neutron/sshkeys.
- The files referenced in key_filename or key_file as seen above will
require those files to be defined as additional files as well.
- Unknown fields in the switch configurations are not allowed. See
[netconf configuration options](https://docs.openstack.org/networking-baremetal/2025.1/configuration/ml2/device_drivers/netconf-openconfig.html) and [generic switch configuration](https://docs.openstack.org/networking-generic-switch/2025.1/configuration.html).
- For generic switch configurations, the device_type field is mandatory.
After the feature is enabled, you can use the sunbeam baremetal sub-command
to manage the deployed nova-ironic shards, Ironic Conductor groups, and
Neutron switch configurations.
## Managing nova-ironic shards
nova-ironic shards will be deployed while enabling the baremetal feature,
as mentioned above. Additional shards can be added through the following
command:
```default
sunbeam baremetal shard add SHARD
```
nova-ironic shards can be removed by running the following command:
```default
sunbeam baremetal shard delete SHARD
```
The following command can be used to list the currently deployed shards:
```default
sunbeam baremetal shard list
```
## Managing Ironic Conductor groups
By default, sunbeam deploys an ironic-conductor-k8s charm with an empty
conductor-group configuration option. Additional Ironic Conductor groups
will be deployed while enabling the baremetal feature, based on the
conductor-groups configuration mentioned above.
Additional Ironic Conductor groups can be added through the following command:
```default
sunbeam baremetal conductor-groups add GROUP-NAME
```
Ironic Conductor Groups can be removed by running the following command:
```default
sunbeam baremetal conductor-groups delete GROUP-NAME
```
The following command can be used to list the currently Ironic Conductor
Groups:
```default
sunbeam baremetal conductor-groups list
```
## Managing Neutron Switch Configurations
netconf and generic Neutron switch configurations will be added while
enabling the baremetal feature, as mentioned above. Additional configurations
can be added through the following command:
```default
sunbeam baremetal switch-config add netconf|generic NAME --config CONFIGFILE [--additional-file ]
```
An existing switch configuration can be updated with the command:
```default
sunbeam baremetal switch-config update netconf|generic NAME --config CONFIGFILE [--additional-file ]
```
#### NOTE
For the add / update sub-commands, multiple additional files can be
specified.
Note that the same restrictions for the switch configurations mentioned above
still apply when adding new ones or updating existing ones.
A switch configuration can be deleted with the following command:
```default
sunbeam baremetal switch-config delete NAME
```
The following command can be used to list the current Neutron switch
configurations and their protocol:
```default
sunbeam baremetal switch-config list
```
## Disabling Baremetal
To disable this feature, run the following command:
```default
sunbeam disable baremetal
```
For information on how to access and use Ironic, check the
[Baremetal feature nodes](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/baremetal-nodes.md) page.
# index.html.md
# Managing TLS
* [Implementing TLS using a third-party CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md)
* [Enable the TLS feature](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md#enable-the-tls-feature)
* [Gather the certificate signing requests](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md#gather-the-certificate-signing-requests)
* [Request TLS certificates](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md#request-tls-certificates)
* [Input TLS certificates](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md#input-tls-certificates)
* [Verify that TLS is active](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md#verify-that-tls-is-active)
* [Managing CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md)
* [Enable TLS CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md#enable-tls-ca)
* [Use TLS CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md#use-tls-ca)
* [Disable TLS CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md#disable-tls-ca)
* [Managing Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md)
* [Prerequisites](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md#prerequisites)
* [Enable TLS Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md#enable-tls-vault)
* [Use TLS Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md#use-tls-vault)
* [Disable TLS Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md#disable-tls-vault)
# index.html.md
# Managing CA
This feature is used to encrypt all cloud service endpoints (both public
and private) using TLS certificates obtained from an external provider.
It does this by interfacing with the existing Traefik instances in the
cloud. A Traefik instance is associated with either public or private
cloud traffic.
## Enable TLS CA
To enable TLS, you’ll need to provide information that identifies your
chosen Certificate Authority. Do this by specifying a CA certificate and
its CA certificate chain.
Run the following command to enable TLS for public endpoints:
```default
sunbeam enable tls ca --ca --ca-chain
```
#### NOTE
Omit the `--ca-chain` option when using self-signed certificates.
To enable TLS for public, internal and rgw endpoints, be explicit by
using the `--endpoint` option:
```default
sunbeam enable tls ca --ca --ca-chain --endpoint public --endpoint internal --endpoint rgw
```
## Use TLS CA
TLS certificates must now be provided to the Traefik units. This is
covered on the [Implement TLS using a third-party CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md) page.
## Disable TLS CA
To disable TLS in the cloud, run the following command:
```default
sunbeam disable tls ca
```
This command removes the manual-tls-certificates charm from being the certificate Authority and all services will work as if TLS was never enabled.
# index.html.md
# Implementing TLS using a third-party CA
This page shows how to implement TLS when using an external Certificate
Authority for your certificates.
#### TIP
For conceptual background on TLS in Canonical OpenStack see the
[Service endpoint encryption](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/service-endpoint-encryption.md) page.
## Enable the TLS feature
### CA
This method relies on the TLS CA feature. See the [TLS CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-ca.md)
feature page for how to enable it.
If the feature is ever disabled (see the feature page), to re-enable,
the entire procedure given below must be repeated.
### Vault
This method relies on the TLS Vault feature. See the [TLS Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/tls-vault.md)
feature page for how to enable it.
If the feature is ever disabled (see the feature page), to re-enable,
the entire procedure given below must be repeated.
## Gather the certificate signing requests
You’ll need the certificate signing requests (CSRs) for each available
Traefik unit.
### CA
To retrieve CSRs for which certificates are not yet provided, run:
```console
sunbeam tls ca list_outstanding_csrs
```
Sample output:
```default
+------------------+------------------------------------------------------------------+
| Unit name | CSR |
+------------------+------------------------------------------------------------------+
| traefik/0 | -----BEGIN CERTIFICATE REQUEST----- |
| | MIICrDCCAZQCAQAwRTEUMBIGA1UEAwwLMTAuMjAuMjEuMTMxLTArBgNVBC0MJDk3 |
| | MmI3YTU3LTM0OTktNDVhNS04OWJkLTM3NjliYzk1MjY1ZTCCASIwDQYJKoZIhvcN |
| | AQEBBQADggEPADCCAQoCggEBAIxYmLNAIxhbIjqQtVNg6faO4rnl1vHrXp9MdmpP |
| | aED4lqq6/Zn+maeVv3Yh6de+GvyZIxXUBRpyZF5Z6qQSIJ4V63ZpCsSPDNUnjEmA |
| | pHnNrAFI87JHvXEBMl+6nhnMJP4b2DsWF0orP8G/zvaMxABzMKlQ4GoKUkz24UJZ |
| | wCrRnsiPiMgKGTW/zNSFgN0wigyFf0gxJTofKWOHv0KRK1H6zBojwZCBwi1x1A6d |
| | 0PhwSz0GxMcrPOkc/Z1cNDg4dySJvm6rn0DLSHE77ZaCgdurS2rrE8WtpPp95E78 |
| | wYRhbcTdLFQTdVkDPClSfYNZK4FjiybgkXq5WTojELt4pscCAwEAAaAiMCAGCSqG |
| | SIb3DQEJDjETMBEwDwYDVR0RBAgwBocEChQVDTANBgkqhkiG9w0BAQsFAAOCAQEA |
| | ZqR2aVYzFD1KvEEFajxJAz8agcpPJougSr9iKEK101/7pQVLDqeCvusJHfv5clYO |
| | RCMxNoAuPFFt83j9V0Sg7FnVLc6ftT9f0C3jWWVbCxZbVMTJ4RcIiYKsjhC8PgpU |
| | J94cQgo4xkcqWc2bpOsEIOyvXgK+AWe9TXhg3EihecDS4Sho7wtDRayR3BL/bOiF |
| | rZGFgnkAgHCNoqHN9IhOqmKm0XWn0XNlP1t6IWih5dGGoYeka135+REKYo4G3kYe |
| | EKqgE3AGkPtjp4nuD33oWa+XK30XPFCRHqdcvenjMfdAPRw+MwAsPWXmihnnSGFh |
| | pVEcQwo0HC3L5LHCVZBdNA== |
| | -----END CERTIFICATE REQUEST----- |
| traefik-public/0 | -----BEGIN CERTIFICATE REQUEST----- |
| | MIICrDCCAZQCAQAwRTEUMBIGA1UEAwwLMTAuMjAuMjEuMTIxLTArBgNVBC0MJDI4 |
| | NzllYjlmLTZhOWUtNGNjMC1hZGIyLWZmOWQ5YzU3ZThiMDCCASIwDQYJKoZIhvcN |
| | AQEBBQADggEPADCCAQoCggEBANAr4HyhL70XlRAeEhc3Xia3dJ8hLtD4hDAzMRc3 |
| | Cd0zdYoKhniZw9Crhp+zdzBwyiVaACj8XiHdl70u7aCts4IJ40GDw4CnWnM5/SHP |
| | I5LYFi6PT4cHQL0SUlIhgaCVMpZQoFJT4TqcS/Wowyh5sl2ZlNDr0OMArHbtUeuG |
| | FQ69cjvMyOxXhMcxPFr21jrXsVLenqJRfTieA7Qev05C9bxJpDcl2CPmTY3ehu0g |
| | evqCkCD3/Kq8H12SFidwQSjip1C//z2Jlg7ndhapf1YXfP6BwrDzF6xxDqExb2Ie |
| | RghC9m3zkNKvIuH4c3MKE6DQsFqf8/LpUMcW7IFyE7R0WgECAwEAAaAiMCAGCSqG |
| | SIb3DQEJDjETMBEwDwYDVR0RBAgwBocEChQVDDANBgkqhkiG9w0BAQsFAAOCAQEA |
| | ixa/O4qFUA69EJRgpTV/Wq/aojIJhBvKZcVt+wbniYo+XUsTbJJCH0v1Ja6p2CYX |
| | uLkRN/NlxetQouAb7Iw8tNXgfxHbje6t+63+f8mmK1eVrJ1euDSdOi/cyyVLz/3H |
| | MWU82Kzdk44EDi+NyQLDQttVJLdGMvME7/W8MNEEj4qYUoMDcbq4CnxS6P37TDO9 |
| | sUwn5Q4Ygju4QH+wWasN0hhln0lc55azYXc7y3KAOee0NZQTAM/QjJkBQ4KoA2Bk |
| | HN0GczVe9vj+8NYMgbdQ5u7b2ZxU1E1hFM/MhQUHP1vJlGVP6znmvojLo2FO07DH |
| | qW/PnbNh7gQuYZOh+zW8+A== |
| | -----END CERTIFICATE REQUEST----- |
+------------------+------------------------------------------------------------------+
```
To get the output in YAML format, use option `--format yaml`.
In this example there are two CSRs - one for unit `traefik/0` (internal traffic)
and one for unit `traefik-public/0` (public traffic).
### Vault
To retrieve CSRs for which certificates are not yet provided, run:
```console
sunbeam tls vault list_outstanding_csrs
```
Sample output:
```default
+------------------+------------------------------------------------------------------+
| Unit name | CSR |
+------------------+------------------------------------------------------------------+
| vault/0 | -----BEGIN CERTIFICATE REQUEST----- |
| | MIIClTCCAX0CAQAwUDEfMB0GA1UEAwwWZGV2b3BzLXNvbHV0aW9ucy5jbG91ZDEt |
| | MCsGA1UELQwkZDhlMzA2OGItOTgzYy00ODU4LWFiYTEtNjhkZGYyMGExNmE2MIIB |
| | IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo4vSTZ/vg3CbFzb1rwbnLZ0O |
| | 6pMu0kcarXwsqfu+nB2Teqv613Zs7+1vGaA9ZyNbo/OquyDsXmBNPeBXAXpXYMmI |
| | RVv6dMDaSOhTbKUYbqSblKhAV+bonHceP9NjFUlFzfcpJSXWFlJeYyQKNGzpgQBf |
| | zyG/oiL0xJjsaM1Ezg5EA3dMnl5ssz3PH/SGHyhuoWytbqEDC5DUcnUEo1tZxEx8 |
| | U4NdQKSFLZVA/pYonR9JeEzdaaqlXSFEOCJe+ktzHGGwXMMhfy4MITVwqr+ILDXD |
| | dEtpDYLuF+GHXyBn2Q7EinuTliPQkt1toNs/1ZDKdiZRHlKg9B0nDO93UorIgQID |
| | AQABoAAwDQYJKoZIhvcNAQELBQADggEBAAOiCoOKiFfGAH4xa9MBvptS53SGg/SH |
| | uqXlN3LyBY2H0Rf9iQp/wZXsKoc/ngEvwQWWx/+isD8mmVo/0v5ar7LIGZScHL7g |
| | n4mG9wlnpf4zYp1KmvP4+RWqmSHsLjicstUlAvcZQaJusZc/reorlGZWp6pXbL/G |
| | 00BFThDc8MCR834Q3mEqJkpQ52gkUL4DxekW0+d56uwvEXaP3++/wZQ8GEdFTnYT |
| | wfS3/inadYtpj5t5vQPJBeMqie47/TVRXUDqkCsZeQiX/VYHxpAlfqpHBrQZklap |
| | 9KdhcRFpBmGy2LlrJYSJcZ7SGNGzHkpsgyAuR3XPV1N5ok9EAmftWMc= |
| | -----END CERTIFICATE REQUEST----- |
+------------------+------------------------------------------------------------------+
```
## Request TLS certificates
Request one TLS certificate for each generated CSR.
### CA
You’ll need to supply the Certificate Authority (identified in the
`enable` command) with the CSRs. Do this via the certificate authority’s web site.
#### NOTE
Ensure the TLS certificate from CA has Subject Alternative Name with IP
Address of the service if DNS names are not used.
### Vault
You’ll need to supply the Certificate Authority (identified in the
`enable` command) with the CSRs. Do this via the certificate authority’s web site.
#### NOTE
The CA certificate needs to be generated as a CA certificate, not as a
regular TLS certificate. Vault will set its `common_name`
configuration option from the hostnames configured for the internal,
RGW, and public ingress endpoints. The CA certificate must have the
same domain defined in the `alt_names` section of the CA
configuration file used to sign the CSR.
## Input TLS certificates
Run the command below to inject the newly acquired TLS certificates into the cloud:
### CA
```console
sunbeam tls ca unit_certs
```
You will be prompted for a TLS certificate for each Traefik unit.
This example’s final total output is:
```default
Base64 encoded Certificate for traefik/0 CSR Unique ID: 9c90972f-ec72-41b9-b6e4-2793ee052531: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...
Base64 encoded Certificate for traefik-public/0 CSR Unique ID: be71a3bd-8d3a-411b-b258-2413d36100ce: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...
CA certs configured
```
Alternatively, to avoid prompts, update TLS certificates in the manifest file
(see the certificates block in [manifest reference](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/manifest-file-reference.md)):
```console
sunbeam tls ca unit_certs --manifest
```
### Vault
```console
sunbeam tls vault unit_certs
```
You will be prompted for a TLS certificate for the Vault unit.
This example’s final total output is:
```default
Base64 encoded Certificate for vault/0 CSR Unique ID: d8e3068b-983c-4858-aba1-68ddf20a16a6: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...
CA certs configured
```
Alternatively, to avoid prompts, update TLS certificates in the manifest file
(see the certificates block in [manifest reference](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/manifest-file-reference.md)):
```console
sunbeam tls vault unit_certs --manifest
```
Once the signed certificate is deployed, Vault will automatically issue TLS certificates for the endpoints specified during the TLS Vault enablement process. These certificates will be used by the Traefik instances to secure the cloud service endpoints.
## Verify that TLS is active
Generate an openrc file:
```console
sunbeam openrc
```
This file should use an HTTPS link for `OS_AUTH_URL` (Keystone) and a value for `OS_CACERT`,
which is the file path to the CA certificate.
### CA
```default
# openrc for access to OpenStack
export OS_USERNAME=admin
export OS_PASSWORD=*******
export OS_AUTH_URL=https://10.20.21.12/openstack-keystone/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_PROJECT_NAME=admin
export OS_AUTH_VERSION=3
export OS_IDENTITY_API_VERSION=3
export OS_CACERT=/home/ubuntu/.config/openstack/ca_bundle.pem
```
### Vault
```default
# openrc for access to OpenStack
export OS_USERNAME=admin
export OS_PASSWORD=*******
export OS_AUTH_URL=https://public.mydomain.com/openstack-keystone/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_PROJECT_NAME=admin
export OS_AUTH_VERSION=3
export OS_IDENTITY_API_VERSION=3
export OS_CACERT=/home/ubuntu/.config/openstack/ca_bundle.pem
```
Generate a cloud-config file:
```console
sunbeam cloud-config --admin --update
```
Similarly, this file should use HTTPS for `auth_url` and have a file for `cacert`:
### CA
```default
clouds:
sunbeam-admin:
auth:
auth_url: https://10.20.21.12/openstack-keystone/v3
password: pS6glK5TQRNf
project_domain_name: admin_domain
project_name: admin
user_domain_name: admin_domain
username: admin
cacert: /home/ubuntu/.config/openstack/ca_bundle.pem
```
### Vault
```default
clouds:
sunbeam-admin:
auth:
auth_url: https://public.mydomain.com/openstack-keystone/v3
password: pS6glK5TQRNf
project_domain_name: admin_domain
project_name: admin
user_domain_name: admin_domain
username: admin
cacert: /home/ubuntu/.config/openstack/ca_bundle.pem
```
Set `OS_CLOUD` to use the credentials generated by `cloud-config`:
```bash
export OS_CLOUD=sunbeam-admin
```
To verify **public** endpoints, run:
```console
openstack endpoint list --interface public
```
The output should use HTTPS for all URLs:
### CA
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| 05dd03b906af463cbbf85164bb4c208a | RegionOne | nova | compute | True | public | https://10.20.21.12:443/openstack-nova/v2.1 |
| 4880f1558ed94739ae9729d638cea95f | RegionOne | cinderv2 | volumev2 | True | public | https://10.20.21.12:443/openstack-cinder/v2/$(tenant_id)s |
| 809d07f8b2e84f49afa2b3ebcabbad03 | RegionOne | cinderv3 | volumev3 | True | public | https://10.20.21.12:443/openstack-cinder/v3/$(tenant_id)s |
| 9c10da39bb2e46588c05018a3098f1aa | RegionOne | neutron | network | True | public | https://10.20.21.12:443/openstack-neutron |
| bfdfca65a8a24e4ebe8340dd169b8012 | RegionOne | glance | image | True | public | https://10.20.21.12:443/openstack-glance |
| cd7490239e6845ffa8c6651300264e5a | RegionOne | keystone | identity | True | public | https://10.20.21.12/openstack-keystone/v3 |
| f7552dc54b4e4d11a1ffa1289957088c | RegionOne | placement | placement | True | public | https://10.20.21.12:443/openstack-placement |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
```
### Vault
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| 05dd03b906af463cbbf85164bb4c208a | RegionOne | nova | compute | True | public | https://public.mydomain.com:443/openstack-nova/v2.1 |
| 4880f1558ed94739ae9729d638cea95f | RegionOne | cinderv2 | volumev2 | True | public | https://public.mydomain.com:443/openstack-cinder/v2/$(tenant_id)s |
| 809d07f8b2e84f49afa2b3ebcabbad03 | RegionOne | cinderv3 | volumev3 | True | public | https://public.mydomain.com:443/openstack-cinder/v3/$(tenant_id)s |
| 9c10da39bb2e46588c05018a3098f1aa | RegionOne | neutron | network | True | public | https://public.mydomain.com:443/openstack-neutron |
| bfdfca65a8a24e4ebe8340dd169b8012 | RegionOne | glance | image | True | public | https://public.mydomain.com:443/openstack-glance |
| cd7490239e6845ffa8c6651300264e5a | RegionOne | keystone | identity | True | public | https://public.mydomain.com/openstack-keystone/v3 |
| f7552dc54b4e4d11a1ffa1289957088c | RegionOne | placement | placement | True | public | https://public.mydomain.com:443/openstack-placement |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
```
To verify **internal** endpoints, run:
```console
openstack endpoint list --interface internal
```
The output should use HTTPS for all URLs:
### CA
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| 04f9fb67ac6d4295a15d19ac829845b1 | RegionOne | neutron | network | True | internal | https://10.20.21.13:443/openstack-neutron |
| 05dda52ae04b424fa7f6083d4a888be2 | RegionOne | glance | image | True | internal | https://10.20.21.13:443/openstack-glance |
| 3fa47154d2c3425d987081600ab6b284 | RegionOne | keystone | identity | True | internal | https://10.20.21.13/openstack-keystone/v3 |
| 6240b34b08cc462a98ab4d37e1ea2770 | RegionOne | placement | placement | True | internal | https://10.20.21.13:443/openstack-placement |
| 6f7ef31c3f994d8a8f66fb749871ff26 | RegionOne | nova | compute | True | internal | https://10.20.21.13:443/openstack-nova/v2.1 |
| a9b1ad2b2e524db5b6147abfcca20eea | RegionOne | cinderv2 | volumev2 | True | internal | https://10.20.21.13:443/openstack-cinder/v2/$(tenant_id)s |
| ef9b8eeb54df468ebfd65adc851092b1 | RegionOne | cinderv3 | volumev3 | True | internal | https://10.20.21.13:443/openstack-cinder/v3/$(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
```
### Vault
```default
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
| 04f9fb67ac6d4295a15d19ac829845b1 | RegionOne | neutron | network | True | internal | https://internal.mydomain.com:443/openstack-neutron |
| 05dda52ae04b424fa7f6083d4a888be2 | RegionOne | glance | image | True | internal | https://internal.mydomain.com:443/openstack-glance |
| 3fa47154d2c3425d987081600ab6b284 | RegionOne | keystone | identity | True | internal | https://internal.mydomain.com/openstack-keystone/v3 |
| 6240b34b08cc462a98ab4d37e1ea2770 | RegionOne | placement | placement | True | internal | https://internal.mydomain.com:443/openstack-placement |
| 6f7ef31c3f994d8a8f66fb749871ff26 | RegionOne | nova | compute | True | internal | https://internal.mydomain.com:443/openstack-nova/v2.1 |
| a9b1ad2b2e524db5b6147abfcca20eea | RegionOne | cinderv2 | volumev2 | True | internal | https://internal.mydomain.com:443/openstack-cinder/v2/$(tenant_id)s |
| ef9b8eeb54df468ebfd65adc851092b1 | RegionOne | cinderv3 | volumev3 | True | internal | https://internal.mydomain.com:443/openstack-cinder/v3/$(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------------+
```
To query for available cloud images, run:
```console
openstack image list
```
This should not result in any errors; the images should be displayed:
```default
+--------------------------------------+--------+--------+
| ID | Name | Status |
+--------------------------------------+--------+--------+
| 01a247e1-74cb-477d-80ca-5d834be8639b | ubuntu | active |
+--------------------------------------+--------+--------+
```
# index.html.md
# Managing Vault
This feature is used to encrypt all cloud service endpoints (both public
and private) using TLS certificates generated by Vault, which acts as an intermediary CA.
It does this by interfacing with the existing Traefik instances in the
cloud. A Traefik instance is associated with either public or private
cloud traffic.
## Prerequisites
To use TLS Vault, you must configure the hostname for Traefik, enable the Vault feature in your cloud and unseal and authorize the Vault charm.
Follow this guide [Enable Vault](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/vault.md).
## Enable TLS Vault
To enable TLS Vault, you’ll need to provide information that identifies your
chosen Certificate Authority. Do this by specifying a CA certificate and
its CA certificate chain.
Run the following command to enable TLS Vault for public endpoints:
```default
sunbeam enable tls vault --ca --ca-chain
```
#### NOTE
Omit the `--ca-chain` option when using self-signed certificates.
To enable TLS Vault for public, internal and rgw endpoints, be explicit by
using the `--endpoint` option:
```default
sunbeam enable tls vault --ca --ca-chain --endpoint public --endpoint internal --endpoint rgw
```
## Use TLS Vault
TLS certificates must now be provided to the Vault unit. This is
covered on the [Implement TLS using a third-party CA](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/managing-tls/implement-tls-using-a-third-party-ca.md) page.
## Disable TLS Vault
To disable TLS Vault in the cloud, run the following command:
```default
sunbeam disable tls vault
```
This command removes the manual-tls-certificates charm and removes Vault from being the intermediary certificate authority, as well as, clear the external hostnames on the corresponding Traefik endpoints. All services will work as if TLS was never enabled.
# index.html.md
# Installation
* [Install Canonical OpenStack using the manual bare metal provider](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-the-manual-bare-metal-provider.md)
* [Install Canonical OpenStack using Canonical MAAS](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/install/install-canonical-openstack-using-canonical-maas.md)
# index.html.md
# Install Canonical OpenStack using the manual bare metal provider
This how-to guide provides all necessary information to install [Canonical OpenStack](https://canonical.com/openstack) with
Sunbeam using the manual bare metal provider.
Make sure you get familiar with the following sections before proceeding with any instructions
listed below:
* [Architecture](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/architecture.md)
* [Design considerations](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/design-considerations.md)
* [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md)
* [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md)
#### NOTE
This how-to guide is intended to serve operators willing to deploy a production-grade cloud.
If you’re looking for some simple learning materials instead, please refer to the
[Tutorials](https://canonical-openstack.readthedocs-hosted.com/2024.1//tutorial/index.md) section of this documentation.
## Requirements
You will need:
* two dedicated physical networks with an unlimited access to the Internet
* one dedicated physical machine with:
* hardware specifications matching minimum hardware specifications for the *Cloud* node as
documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* fresh Ubuntu Server 24.04 LTS installed
If you can’t provide an unlimited access to the Internet, see the
[Manage a proxied environment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-a-proxied-environment.md) section.
Additional machines can be added later. See the [Scaling the cluster out](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md) how-to guide.
## Install Canonical OpenStack
When using the manual bare metal provider, Canonical OpenStack installation process is
relatively simple and takes around 30 minutes to complete, depending on your Internet connection
speed.
#### WARNING
Canonical Juju does not yet support controller HA modeling capabilities when deployed on top
Kubernetes. This means that Canonical OpenStack clouds deployed using the manual bare metal
provider do not provide HA for all types of governance functions by default. To bypass this
limitation Canonical recommends [using an external highly available Juju controller](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/using-an-existing-juju-controller.md). External
controller has to be registered before running the `sunbeam cluster bootstrap` command.
### Install the snap
First, install the `openstack` snap:
```text
sudo snap install openstack
```
This will install the latest stable version by default. You can use the `--channel` switch to
install a different version of OpenStack instead.
To list all available versions, execute the following command:
```text
snap info openstack
```
### Prepare the machine
To prepare the machine for Canonical OpenStack usage, execute the following command:
```text
sunbeam prepare-node-script --bootstrap | bash -x && newgrp snap_daemon
```
This command will:
* ensure all required software dependencies are installed, including the `openssh-server`,
* configure passwordless access to the `sudo` command for all terminal commands for the
currently logged in user (i.e. `NOPASSWD:ALL`).
Alternatively, you can let Sunbeam generate a script that you can further review and execute
step by step:
```text
sunbeam prepare-node-script --bootstrap
```
### Bootstrap the cloud
To bootstrap the cloud, execute the following command:
```text
sunbeam cluster bootstrap --role control,compute,storage
```
This will assign all roles (`control`, `compute`, `storage`) to the machine by default.
You can use the `--role` switch to narrow them down. See the [Architecture](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/architecture.md) section for more
details.
#### NOTE
A node can also be bootstrapped with the `network` role assigned.
When prompted, answer some interactive questions. Below is a sample output from the *cloud-1*
machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section:
```text
Management network (172.16.1.0/24): 172.16.1.0/24
Use proxy to access external network resources? [y/n] (n): n
Enter database toplogy: single/multi (cannot be changed later) (single): single
Enter a region name (cannot be changed later) (RegionOne): RegionOne
OpenStack APIs IP ranges (172.16.1.201-172.16.1.240): 172.16.1.201-172.16.1.240
Ceph devices (/dev/disk/by-id/wwn-0x500a0751e86b8eee): /dev/sdb
```
You can also refer to the [Interactive configuration prompts](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/interactive-configuration-prompts.md) section for detailed description of
each of those questions and some examples.
#### NOTE
The `network` role is mutually exclusive with the `compute` role and cannot be assigned
to the same machine. See the [Architecture](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/architecture.md) section for more
details.
Also note that answers to all those questions can be automated with the use of a
[Deployment manifest](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/deployment-manifest.md).
One finished, you should be able to see the following message on your screen:
```text
Node has been bootstrapped with roles: storage, compute, control
```
### Configure the cloud
Finally, configure the cloud for sample usage:
```text
sunbeam configure
```
Unless directed otherwise, this command will create sample project and user account. You can use
the `--openrc` switch to automatically generate an OpenStack RC file for this user (e.g.
`--openrc my-openrc`).
When prompted, answer some interactive questions. Below is a sample output from the *cloud-1*
machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section:
```text
Local or remote access to VMs [local/remote] (local): remote
External network (172.16.2.0/24): 172.16.2.0/24
External network's gateway (172.16.2.1): 172.16.2.1
External network's allocation range (172.16.2.2-172.16.2.254): 172.16.2.2-172.16.2.254
External network's type [flat/vlan] (flat): flat
Populate OpenStack cloud with demo user, default images, flavors etc [y/n] (y): y
Username to use for access to OpenStack (demo): demo
Password to use for access to OpenStack (IY********):
Project network (192.168.0.0/24): 192.168.0.0/24
Project network's nameservers (172.16.1.11 8.8.8.8 172.16.1.1 192.168.2.22 172.16.1.14): 8.8.8.8
Enable ping and SSH access to instances? [y/n] (y): y
External network's interface [eno2] (eno2): eno2
```
You can also refer to the [Interactive configuration prompts](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/interactive-configuration-prompts.md) section for detailed description of
each of those questions and some examples.
Also note that answers to all those questions can be automated with the use of a
[Deployment manifest](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/deployment-manifest.md).
One finished, you should be able to see the following message on your screen:
```text
The cloud has been configured for sample usage.
You can start using the OpenStack client or access the OpenStack dashboard at http://172.16.1.203:80/openstack-horizon
```
Note that the IP address of the OpenStack dashboard (here `172.16.1.203`) might be different
in your environment.
## Related how-to guides
Now that Canonical OpenStack is installed, you might want to check out the following how-to guides:
* [Using the OpenStack dashboard](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/using-the-openstack-dashboard.md)
* [Using the OpenStack client](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/using-the-openstack-cli.md)
* [Scaling the cluster out](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md)
# index.html.md
# Install Canonical OpenStack using Canonical MAAS
This how-to guide provides all necessary information to install [Canonical OpenStack](https://canonical.com/openstack) with
Sunbeam using [Canonical MAAS](https://maas.io/).
Make sure you get familiar with the following sections before proceeding with any instructions
listed below:
* [Architecture](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/architecture.md)
* [Design considerations](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/design-considerations.md)
* [Network traffic isolation with MAAS](https://canonical-openstack.readthedocs-hosted.com/2024.1//explanation/network-traffic-isolation-with-maas.md)
* [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md)
* [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md)
#### NOTE
This how-to guide is intended to serve operators willing to deploy a production-grade cloud.
If you’re looking for some simple learning materials instead, please refer to the
[Tutorials](https://canonical-openstack.readthedocs-hosted.com/2024.1//tutorial/index.md) section of this documentation.
## Requirements
You will need:
* at least two dedicated physical networks with an unlimited access to the Internet
* one (or at least three for full HA) dedicated physical machine(s) with:
* hardware specifications matching minimum hardware specifications for the *Cloud* node as
documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* one (or at least three for full HA) dedicated physical machine(s) with:
* hardware specifications matching minimum hardware specifications for the *Governor* node as
documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* fresh Ubuntu Server 24.04 LTS installed
* one (or at least three for full HA) dedicated virtual machine(s), running on the *Governor*
node(s), with:
* hardware specifications matching minimum hardware specifications for the *MAAS* node as
documented in the [Canonical MAAS installation requirements](https://maas.io/docs/installation-requirements).
* fresh Ubuntu Server 24.04 LTS installed
* one dedicated virtual machine(s), running on a *Governor* node, with:
* hardware specifications matching minimum hardware specifications for the *Sunbeam Client* node
as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* fresh Ubuntu Server 24.04 LTS installed
* one (or at least three for full HA) dedicated virtual machine(s), running on the *Governor*
node(s), with:
* hardware specifications matching minimum hardware specifications for the *Sunbeam Controller*
node as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
* one (or at least three for full HA) dedicated virtual machine(s), running on the *Governor*
node(s), with:
* hardware specifications matching minimum hardware specifications for the *Juju Controller*
node as documented under the [Enterprise requirements](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/enterprise-requirements.md) section
If you can’t provide unlimited access to the Internet, see the [Manage a proxied
environment](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/misc/manage-a-proxied-environment.md) page.
Additional machines can always be added later. See the [Scaling the cluster out](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/operations/scaling-the-cluster-out.md) how-to guide.
## Install Canonical OpenStack
The following section assumes a generic knowledge of OpenStack and Canonical MAAS. Please refer to
the upstream [OpenStack documentation](https://docs.openstack.org/) and [MAAS documentation](https://maas.io/docs)
for more information.
### Pick a deployment name
Before you get started you have to pick a name for your Canonical OpenStack deployment. This
name will be used in various parts of this how-to guide. We’ll refer to it as a *deployment name*.
-internal-api`, and
`-public-api` accordingly where the `` prefix matches the deployment name.
Depending on the number of optional features being used, you have to account for around 10-20
IP addresses per each range.
If the cloud is intended to use feature [Instance recovery](https://canonical-openstack.readthedocs-hosted.com/2024.1//how-to/features/instance-recovery.md),
new range have to be created under `storage` cloud networks with label `-storage-ippool`.
Single IP address is sufficient for this range.
Reserved IP ranges from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section would look like as follows:
Refer to [MAAS documentation](https://maas.io/docs) for more information on creating reserved IP ranges.
#### Enlist, commission and configure machines
All machines but the Governor and Sunbeam Client nodes must be enlisted, commissioned and
configured in MAAS.
In the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section sunbeam-controller-1, sunbeam-controller-2, sunbeam-controller-3,
juju-controller-1, juju-controller-2 and juju-controller-3 machines are VMs running on
Governor nodes.
##### Assign machine tags
In addition to some generic settings operators must assign machine tags to all nodes that they
intend to use in their deployment.
Please refer to the following table for information on which machine tags to assign to which nodes
in the cluster:
#### Tab. 1. Machine tags assignment.
| Machine tag | Purpose | Nodes to assign the tag to | Required cloud networks |
|-------------------|-------------------------------------------------------------|-----------------------------------------------------------------------|------------------------------------------------|
| openstack- | Defines which machines to use in this particular deployment | Cloud, Control, Compute, Storage, Sunbeam Controller, Juju Controller | None |
| control | Defines where to host cloud control functions | Cloud, Control | data, internal, management, public, storage |
| region-controller | Defines where to host cloud region controller functions | Region Controller | internal, management, public |
| compute | Defines where to host cloud compute functions | Cloud, Compute | data, internal, management, storage |
| storage | Defines where to host cloud storage functions | Cloud, Storage | internal, management, storage, storage-cluster |
| network | Defines where to host cloud network functions | Cloud, Network | internal, management, data |
| sunbeam | Defines where to host the Sunbeam controller | Sunbeam Controller | management |
| juju-controller | Defines where to host the Juju controller | Juju Controller | management |
Note that the `` suffix must match the deployment name.
Machines from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md) section would look like as follows:
Refer to [MAAS documentation](https://maas.io/docs) for more information on assigning machine tags.
##### Configure network
In addition to configuring network interfaces attached to the Generic physical network (or any
other physical networks if using more than one for traffic segmentation purposes), operators must
also configure the network interface attached to the External physical network. This is done by
leaving the *Subnet* field of this interface as *Unconfigured* and assigning the
`neutron:physnet1` network tag.
For example, network configuration of the *cloud-1* machine from the [Example physical configuration](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md)
section would look like as follows:
Refer to [MAAS documentation](https://maas.io/docs) for more information on assigning network tags.
##### Configure storage
All storage devices that are expected to serve as Ceph OSDs must have the `ceph` storage tag
assigned.
In the example configuration those would be `/dev/sdb` devices on *cloud-1*, *cloud-2* and
*cloud-3* machines.
Refer to [MAAS documentation](https://maas.io/docs) for more information on assigning storage tags.
### Install the snap
#### NOTE
All terminal commands used in this how-to guide are run from the first *Sunbeam Client* machine
(aka primary node).
First, install the `openstack` snap:
```text
sudo snap install openstack
```
This will install the latest stable version by default. You can use the `--channel` switch to
install a different version of OpenStack instead.
To list all available versions, execute the following command:
```text
snap info openstack
```
### Prepare the machine
To prepare the machine for Canonical OpenStack usage, execute the following command:
```text
sunbeam prepare-node-script --client | bash -x
```
This command will:
* install the Juju client,
* create any necessary data directories.
Alternatively, you can let Sunbeam generate a script that you can further review and execute step
by step:
```text
sunbeam prepare-node-script --client
```
### Add the Canonical MAAS provider
By default Sunbeam doesn’t know how to talk to Canonical MAAS. Therefore, information about the
Canonical MAAS provider have to be provided by the operator first.
In order to add the Canonical MAAS provider, execute the `sunbeam deployment add` command:
```text
sunbeam deployment add maas NAME TOKEN URL
```
`NAME` is the deployment name.
`TOKEN` is the MAAS API key.
`URL` is the MAAS URL.
For example, to add the Canonical MAAS provider from the [Example configuration section](https://canonical-openstack.readthedocs-hosted.com/2024.1//reference/example-physical-configuration.md), execute
the following command:
```text
sunbeam deployment add maas mycloud Nehk886eajph68tGEK:HcaG27ACee2X2LuPA2:2GtynUxLHXWmQsRYznKahfy3F6D8e4ex http://172.16.1.14:5240/MAAS
```
mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 46:fc:d8:8d:05:49 brd ff:ff:ff:ff:ff:ff
inet 10.20.20.1/24 scope global br-ex
valid_lft forever preferred_lft forever
inet6 fe80::44fc:d8ff:fe8d:549/64 scope link
valid_lft forever preferred_lft forever
BR_EX_MAC="46:fc:d8:8d:05:49"
BR_EX_IP="10.20.20.1"
```
OpenFlow assigns each port a number so the next step is to find what
number has been assigned to the br-ex port on the br-ex bridge:
```text
sudo openstack-hypervisor.ovs-vsctl get Interface br-ex ofport
65534
PORT_BR_EX=65534
```
Next, gather data about the destination of the request. The IP address
that was pinged earlier was 10.20.20.138:
```text
GUEST_FLOATING_IP="10.20.20.138"
```
The demo-router is going to handle this traffic so the destination MAC
address in this case is actually the MAC address of the demo-routers
port on the external network:
```text
openstack port list --router demo-router
+--------------------------------------+----------------------+-------------------+------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+----------------------+-------------------+------------------------------------------------------------------------------+--------+
| 56a18b9e-07d4-4249-b28b-b6446961a587 | demo-router-floating | fa:16:3e:23:60:97 | ip_address='10.20.20.239', subnet_id='14c63eaf-eeb7-476d-a99d-0a05f6a674f8' | ACTIVE |
| ae7b9a8e-48e8-4c3a-9ef0-710ccba00776 | demo-router-internal | fa:16:3e:70:93:8c | ip_address='192.168.122.1', subnet_id='17e394f9-e12c-4f31-a269-62ddf3308fc8' | ACTIVE |
+--------------------------------------+----------------------+-------------------+------------------------------------------------------------------------------+--------+
ROUTER_EXT_MAC="fa:16:3e:23:60:97"
```
Since this is going to trace a single packet, information about the type
of packet is needed. In this case, it is the echo request which is part
of the ping. An IPv4 ICMP echo request has an `icmp_type` of 8 and a code
of 0. Lastly, `nw_ttl` needs to be set to accommodate the number of hops
needed. In this case 64 is a reasonable value.
Putting this all together:
```text
sudo openstack-hypervisor.ovs-appctl ofproto/trace \
br-ex \
icmp,\
in_port=$PORT_BR_EX,\
dl_src=$BR_EX_MAC,\
dl_dst=$ROUTER_EXT_MAC,\
nw_src=$BR_EX_IP,\
nw_dst=$GUEST_FLOATING_IP,\
nw_ttl=64,\
icmp_type=8,\
icmp_code=0
```
This produces a large amount of output - details of how the packet is
traversing the OpenFlow rules - but the important piece is at the end:
```text
...
65. reg15=0x3,metadata=0x2, priority 100, cookie 0x3d326af3
output:2
```
This shows that the packet was sent out of OpenFlow port number 2. This
corresponds to the intended guest (see “Resolving OpenFlow port numbers”
below).
Finally, delete the security group rule that is permitting ICMP traffic
and check that the trace command now drops the traffic.
```text
openstack security group list --project demo
+--------------------------------------+---------+------------------------+----------------------------------+------+
| ID | Name | Description | Project | Tags |
+--------------------------------------+---------+------------------------+----------------------------------+------+
| 00aed662-f303-47fa-82a7-86cde90a4ee1 | default | Default security group | b8c896d15bb247448edd2d97f7d99f1f | [] |
+--------------------------------------+---------+------------------------+----------------------------------+------+
openstack security group rule list --ingress --protocol icmp 00aed662-f303-47fa-82a7-86cde90a4ee1
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Direction | Remote Security Group | Remote Address Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+
| 33237298-6052-45d9-9a7e-1fee0a7587b7 | icmp | IPv4 | 0.0.0.0/0 | | ingress | None | None |
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+
openstack security group rule delete 33237298-6052-45d9-9a7e-1fee0a7587b7
```
This time the trace command ends with:
```text
...
44. ip,reg0=0x200/0x200,reg15=0x3,metadata=0x2, priority 2001, cookie 0x5eeee244
drop
```
## Resolving OpenFlow port numbers
When looking at OpenFlow rules or tracing a packet, the ports are given
numbers. These are the OpenFlow port numbers. For example, to find what
port 2 corresponds to:
```text
sudo openstack-hypervisor.ovs-vsctl find interface ofport=2 | grep -E "^name"
name : tapd8174cec-c5
```
Often the first part of the corresponding port’s UUID is included in the
name of the device. This enables it to be traced back:
```text
openstack port list | grep d8174cec-c5
| d8174cec-c5ae-4bd0-abb4-9420c3b87e76 | rapid-owl-internal | fa:16:3e:dd:8f:4d | ip_address='192.168.122.83', subnet_id='17e394f9-e12c-4f31-a269-62ddf3308fc8' | ACTIVE |
```
# index.html.md
:
https_proxy: :
no_proxy: ,,...
# Configure OVS DPDK datapath (userspace), improving network performance.
dpdk:
# If enabled, OVS bridges are configured to use the netdev (DPDK) datapath
# instead of the standard system datapath.
enabled: [ true, false ]
# The number of CPU cores to allocate for OVS control plane processing.
control_plane_cores: 1
# The number of CPU cores to allocate for OVS data plane processing.
dataplane_cores: 1
# The amount of hugepage memory (MB) to reserve for OVS DPDK.
memory: 1024
# The DPDK compatible driver that will be assigned to physical
# interfaces connected to the DPDK dapapath.
driver: vfio-pci
# A list of physical interfaces to use with DPDK for each node.
#
# The interfaces will be persistently bound to the configured DPDK
# compatible driver, no longer being visible to the host.
#
# OVS bridges containing those interfaces (or bonds) are expected to
# be defined through Netplan or MAAS. Canonical Openstack will move
# the corresponding Netplan configuration to OVS, using the resulting
# OVS DPDK physical ports.
#
# Example:
# ports:
# r740-dc1-ceph.maas:
# - eno2
# - enp94s0
ports:
:
-
-
# This section defines PCI passthrough configuration, allowing SR-IOV
# VFs, GPUs and other PCI devices to be exposed to Openstack instances.
pci:
# A list of PCI filters specifying which devices to expose.
# The specs can contain exact PCI addresses, address wildcards,
# address regular expressions or vendor/product id tuples.
# SR-IOV interfaces may also contain a Neutron physical network,
# usually known as "physnet".
#
# See the Nova documentation for more details.
# https://docs.openstack.org/nova/latest/configuration/config.html#pci.device_spec
#
# Note that the following list applies to all compute nodes, use
# the "excluded_devices" field to define per-node exclusion lists.
#
# Example:
# device_specs:
# - address: "0000:1b:00.0"
# vendor_id: "8086"
# product_id: "1563"
# physical_network: "physnet1"
device_specs: [ ]
# Per-node PCI device exclusion list, containing excluded PCI addresses.
#
# Example:
# excluded_devices:
# r740-dc1-ceph.maas:
# - "0000:19:00.0"
# - "0000:19:00.1"
excluded_devices:
:
-
-
# A list of aliases that can be used to request PCI devices through
# Nova flavor extra specs.
#
# See the Nova documentation for more details.
# https://docs.openstack.org/nova/latest/configuration/config.html#pci.alias
#
# Example:
# aliases:
# - vendor_id: "8086"
# product_id: "1565"
# device_type: type-VF
# name: "intel-vf"
aliases: { }
bootstrap:
# Management networks shared by hosts
management_cidr: ,,...
# Example:
# management_cidr: 192.168.29.0/24
# Enter database toplogy: single/multi (cannot be changed later)
# This will configure number of databases, single for entire cluster or multiple databases with one per openstack service.
database: single
# Enter a region name (cannot be changed later)
region:
# Example:
# region: RegionOne
k8s-addons:
# Load balancer ranges
loadbalancer: ,,...
user:
# Populate OpenStack cloud with demo user, default images, flavors etc
run_demo_setup: [ true,false ]
# Username to use for access to OpenStack
username:
# Password to use for access to OpenStack
password:
# Network to use for initial project network
cidr:
# Nameservers that guests should use for DNS resolution
nameservers: ...
# Enable ping and SSH access to instances
security_group_rules: [ true,false ]
# Local or remote access to VMs
# Local mode - single node only
remote_access_location: [ local,remote ]
# Name of the physical network to populate the demo project network
# external routing
physnet:
# External networking (deprecated, use external-networks instead)
external_network:
nic: # deprecated
nics:
:
# Examples:
# sunbeam-1.localdomain: enp5s0
# sunbeam-2.localdomain: enp8s0
# sunbeam-3.localdomain: eno3
# CIDR of OpenStack external network
cidr:
# IP address of default gateway for external network
gateway:
# External network's allocation range
range:
# Network type for access to external network
network_type: [ flat,vlan ]
# VLAN ID if 'vlan' is chosen above
segmentation_id:
# External networking
external-networks:
:
nics:
:
# Examples:
# sunbeam-1.localdomain: enp5s0
# sunbeam-2.localdomain: enp8s0
# sunbeam-3.localdomain: eno3
# CIDR of OpenStack external network
cidr:
# IP address of default gateway for external network
gateway:
# External network's allocation range
range:
# Network type for access to external network
network_type: [ flat,vlan ]
# VLAN ID if 'vlan' is chosen above
segmentation_id:
# MicroCeph
microceph_config:
# Disks to attach to MicroCeph nodes
:
osd_devices: ,,...
# WARNING: This will wipe ALL devices listed on the
# dangerous_i_acknowledge_i_will_lose_data_wipe_disks: true
# Examples:
# sunbeam-1.localdomain:
# osd_devices: /dev/vdc,/dev/vdd
# sunbeam-2.localdomain:
# osd_devices: /dev/vdc,/dev/vdd
# dangerous_i_acknowledge_i_will_lose_data_wipe_disks: true
# sunbeam-3.localdomain:
# osd_devices: /dev/vdc,/dev/vdd
endpoints:
# Ips must be part of management_cidr defined above
# or public/internal spaces in MAAS deployments
ingress-internal: # optional
ip: # optional
hostname: # optional
ingress-public: # optional
ip: # optional
hostname: # optional
ingress-rgw: # optional
ip: # optional
hostname: # optional
# Examples:
# ingress-internal:
# hostname: internal.openstack.example.com
# ingress-public:
# ip: 192.168.29.27
# hostname: public.openstack.example.com
# ingress-rgw:
# ip: 192.168.29.28
# Openstack Dashboard Customization
horizon:
resources:
custom_theme:
software:
juju:
bootstrap_args:
-
-
- ...
# Examples:
# - --debug
# - --agent-version=3.2.4
# - --model-default=test-mode=true
# - --model-default=logging-config==INFO;unit=DEBUG
charms:
:
channel:
revision:
config:
:
:
# ...
# Examples:
# keystone-k8s:
# channel: 2024.1/candidate
# glance-k8s:
# channel: 2024.1/candidate
# revision: 66
# config:
# debug: true
# pool-type: replicated
# Special cases
# Configure mysql storage in single mysql scenario
# mysql-k8s:
# storage:
# database:
# Configure mysql storage in multi mysql scenario
# mysql-k8s:
# storage-map:
# keystone-k8s:
# database:
# glance-k8s:
# database:
# ...
# Configure mysql configs in multi mysql scenario
# mysql-k8s:
# config-map:
# keystone-k8s:
# :
# glance-k8s:
# :
# ...
# Configure glance image repository for local storage
# glance-k8s:
# storage:
# local-repository:
# Configure traefik configs per application
# traefik-k8s:
# config-map:
# traefik:
# tls-ca:
# tls-cert:
# tls-key:
# traefik-public:
# tls-ca:
# tls-cert:
# tls-key:
# traefik-rgw:
# tls-ca:
# tls-cert:
# tls-key:
# Configure cinder-volume configs per application
# cinder-volume:
# config-map:
# cinder-volume:
# rabbit-user:
# cinder-volume-noha:
# rabbit-user:
terraform:
:
source:
# Example:
# hypervisor-plan:
# source: /home/ubuntu/deploy-openstack-hypervisor
features:
observability:
embedded:
# Storage for embedded COS can be configured before deployment.
# Resizing these storage options after deployment is not currently
# supported.
software:
charms:
:
storage:
:
# Examples:
# prometheus-k8s:
# storage:
# database: "40G" # default: 20G
# loki-k8s:
# storage:
# active-index-directory: "4G" # default: 2G
# loki-chunks: "10G" # default: 5G
# grafana-k8s:
# storage:
# database: "2G" # default: 1G
# alertmanager-k8s:
# storage:
# data: "2G" # default: 1G
# opentelemetry-collector-k8s:
# storage:
# persisted: "8G" # default: 1G
# storage-map:
# opentelemetry-collector:
# persisted: "2G"
# opentelemetry-collector-infra:
# persisted: "3G"
loadbalancer:
config:
# Enable the Octavia Amphora VM-based load-balancer backend.
# Requires the microovn-sdn and loadbalancer-amphora feature gates.
amphora_enabled: [ true, false ]
# Glance tag used by Octavia to locate the Amphora VM image.
# An image with this tag must exist in Glance before Octavia can
# create load-balancer instances.
amp_image_tag: octavia-amphora
# If true, Sunbeam downloads the upstream Octavia Amphora image from
# tarballs.opendev.org and uploads it to Glance with amp_image_tag.
# Set to false if you already have a suitable image in Glance.
autocreate_image: [ true, false ]
# If true, Sunbeam creates a dedicated Nova flavor for Amphora VM
# instances automatically. Set to false to supply your own flavor ID.
autocreate_flavor: [ true, false ]
# Nova flavor ID for Amphora VM instances.
# Required only when autocreate_flavor is false.
amp_flavor_id:
# If true, Sunbeam creates the Octavia lb-mgmt network and subnet
# automatically using an IPv6 ULA subnet (fd00:a9fe:a9fe::/64).
# Set to false to supply your own network and subnet IDs.
autocreate_network: [ true, false ]
# Neutron network ID for the Octavia lb-mgmt management network.
# Required only when autocreate_network is false.
lb_mgmt_network_id:
# Neutron subnet ID within the lb-mgmt network.
# Required only when autocreate_network is false.
lb_mgmt_subnet_id:
# If true, Sunbeam creates the Neutron security groups for Amphora VM
# ports automatically. Set to false to supply your own group IDs.
autocreate_securitygroups: [ true, false ]
# List of Neutron security group IDs for Amphora VM ports
# (passed to the Octavia charm as amp-secgroup-list).
# Required only when autocreate_securitygroups is false.
lb_mgmt_secgroup_ids:
-
# Neutron security group ID for the Octavia health manager port
# (lb-health-mgr-sec-grp).
# Required only when autocreate_securitygroups is false.
lb_health_secgroup_id:
# Octavia Amphora TLS certificates, keyed by CSR x500UniqueIdentifier.
# Run 'sunbeam loadbalancer list_outstanding_csrs' to obtain subjects.
# Two entries are required: one for amphora-controller-cert (leaf cert)
# and one for amphora-issuing-ca (CA cert).
certificates:
:
# Base64-encoded signed certificate (PEM).
# For amphora-issuing-ca this must be a CA certificate
# (basicConstraints: CA:TRUE).
certificate:
# Base64-encoded CA certificate that signed the certificate above.
ca_certificate:
# Base64-encoded full CA chain (intermediate + root CAs).
# Leave empty if the CA certificate is self-signed.
ca_chain:
software:
charms:
octavia-k8s:
channel:
revision:
config:
:
multus:
channel:
revision:
openstack-port-cni-k8s:
channel:
revision:
manual-tls-certificates:
channel:
revision:
tls:
ca:
config:
# TLS
certificates:
:
# Base64 encoded certificate for unit CSR Unique ID: subject
certificate:
vault:
config:
# TLS
certificates:
:
# Base64 encoded certificate for unit CSR Unique ID: subject
certificate:
storage:
# Storage is keyed by backend type, then by instance name.
# Current backend types are dellsc, hitachi, and purestorage.
dellsc:
:
config:
san-ip:
san-login:
san-password:
protocol: [ fc, iscsi ]
# Shared storage config fields.
volume-backend-name:
backend-availability-zone:
# Additional Dell Storage Center options also use kebab-case.
# Same structure as core.software.
software: { }
hitachi:
:
config:
hitachi-storage-id:
hitachi-pools: ,