Configuring GPU Passthrough

Overview

The GPU passthrough allows full access and direct control of physical GPU device in guests. The guests should have the corresponding drivers to use the devices.

Prerequisites

1. Enable the VT-d settings in BIOS.

2. Add the following kernel parameters.

intel_iommu=on amd_iommu=on

Manual mode

Canonical Openstack will determine if there are any GPU devices. PCI device classes of type Display Controller (0x03) and Processing Accelerators (0x1200) are filtered as GPU devices. The devices are automatically added to Nova PCI whitelist and no user intervention is required.

Maas mode

Maas mode works similar to Manual mode and the detected GPU devices are added to Nova PCI whitelist with no user intervention.

Ensure that MAAS is configured to apply the necessary kernel parameters.

Manifest configuration

Arbitrary PCI devices may be whitelisted through the Canonical Openstack manifest.

Example:

pci:
  device_specs:
    - address: "0000:4b:00.0"
      vendor_id: "10de"
      product_id: "1db4"
  excluded_devices:
    r740-dc1-ceph.maas:
      - "0000:19:00.0"
      - "0000:19:00.1"
      - "0000:1b:00.1"
      - "0000:5e:00.0"
  aliases:
    - vendor_id: "10de"
      product_id: "1db4"
      device_type: type-PCI
      name: "nvidia-gpu"

The device spec filters are highly flexible and can contain PCI address wildcards or PCI vendor/product IDs. See the Nova device spec reference for more details.

The device whitelist will be applied to all the compute nodes. If needed, use the exclusion list to define per-node lists of devices that should not be exposed to Openstack instances.

Configured PCI device aliases may be requested through Nova flavor extra specs.

Attaching GPUs to Openstack instances

Create a flavor with pci_passthrough:alias property.

In the below example, the property is set on an existing flavor.

openstack flavor set m1.tiny --property "pci_passthrough:alias"="nvidia-gpu:1"

Launch a demo instance:

sunbeam launch --name test

Verify the Libvirt domain:

$ sudo openstack-hypervisor.virsh dumpxml instance-00000001 | grep "hostdev mode='subsystem' type='pci'" -A 7
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
  </source>
  <alias name='hostdev0'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>

Verify the PCI device in the guest:

$ ssh -i /home/ubuntu/snap/openstack/x1/sunbeam ubuntu@172.16.2.115 sudo lspci -nn
...
...
04:00.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] [10de:1db4] (rev a1)
...
...

Above example shows the passthrough device Nvidia 3D controller in the guest.