QEMU/KVM Intel ARCA380 GPU passthrough to Linux VM working

Post Reply
zemerdon
Site Admin
Posts: 310
Joined: Mon Jan 23, 2023 8:13 pm

QEMU/KVM Intel ARCA380 GPU passthrough to Linux VM working

Post by zemerdon »

host: Debian 12
kernel: 6.12.33+deb12-amd64
QEMU emulator version 10.0.0 (Debian 1:10.0.0+ds-2~bpo12+2)

VM: ubuntu server 24.10
kernel: 6.11.0-29-generic

get GPU info from lspci -vnn

Code: Select all

69:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Device [172f:4017]
        Flags: bus master, fast devsel, latency 0, IRQ 197, NUMA node 0, IOMMU group 11
        Memory at d5000000 (64-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at d6000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

get driver info from driverctl

Code: Select all

0000:69:00.0 vfio-pci
0000:6a:00.0 (none)
HOST CONFIG

append /etc/initramfs-tools/modules (load vfio modules into initramfs for takeover)

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
append /etc/modules (not sure if this actually works :)

Code: Select all

vfio
vfio_pci
create file /etc/modules-load.d/vfio.conf (gives GPU to vfio before anyone else (i915 driver kept taking it VERY early during boot))

Code: Select all

options vfio-pci ids=8086:56a5 disable_vga=1
softdep i915 pre: vfio-pci
** side note A** the following is a kernel parameter only, for multiple devices

Code: Select all

vfio-pci.ids=vendor_id:device_id,vendor_id:device_id_2
** side note B** multiple devices in a vfio.conf

Code: Select all

options vfio-pci ids=aaaa:1111, bbbb:2222
apply module settings to all kernels

Code: Select all

sudo update-initramfs -u -k all
reboot host.

QEM/KVM CONFIG

creating a VM in virt-manager i just did some standard stuff.

ubuntu server iso, 200gb image, passed a port from my quad NIC for network, left everything alone for the initial OS install.

once ubuntu server was installed, i made sure network was up and i could SSH in plus did a dist-upgrade.

Code: Select all

apt install linux-headers-$(uname -r) intel-gpu-tools libdrm-dev cifs-utils linux-firmware apt-file driverctl
reboot VM, check everything works like SSH etc then shutdown for virt-manager XML editing.

on the host, open the VM XML editor.

i followed this guide: https://gitlab.com/risingprismtv/single ... to-your-VM

removed all the spice stuff, i had to play around a bit because QEMU gave spice dependency errors when trying to save the XML about wanting other spice drivers so i just started removing stuff until it saved.

once the XML was saved and spice display was removed i then added the GPU only. saved and booted the VM. VM booted fine, SSH worked fine, driverctl confirmed i915 driver for the GPU bus. Plugged a monitor in to the GPU and the console login popped right up (this was no longer visible in the virt-manager).

** PROBLEM** but not really The VM isn't taking direct input via virt-manager which i suspect i haven't configured the virt-manager/XML to do so yet, but im not phased because this VM is only for Plex.

XML code thus far...

Code: Select all

<domain type="kvm">
  <name>ubuntu22.10</name>
  <uuid>08070f88-05aa-44e2-bf41-e7e08f9bc934</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">12582912</memory>
  <currentMemory unit="KiB">12582912</currentMemory>
  <vcpu placement="static">8</vcpu>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-10.0">hvm</type>
    <firmware>
      <feature enabled="yes" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" secure="yes" type="pflash" format="raw">/usr/share/OVMF/OVMF_CODE_4M.ms.fd</loader>
    <nvram template="/usr/share/OVMF/OVMF_VARS_4M.ms.fd" templateFormat="raw" format="raw">/var/lib/libvirt/qemu/nvram/ubuntu22.10_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/ubuntu22.10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <tpm model="tpm-crb">
      <backend type="emulator" version="2.0"/>
    </tpm>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb3" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x69" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <watchdog model="itco" action="reset"/>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
    <rng model="virtio">
      <backend model="random">/dev/urandom</backend>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </rng>
  </devices>
</domain>
VM dmesg output... (i think 403 was when i plugged in the monitor :)

Code: Select all

root@plex:/home# dmesg | grep i915
[    5.430023] i915 0000:07:00.0: [drm] Found DG2/G11 (device ID 56a5) display version 13.00 stepping C0
[    5.431296] i915 0000:07:00.0: [drm] VT-d active for gfx access
[    5.446481] i915 0000:07:00.0: vgaarb: deactivate vga console
[    5.465200] i915 0000:07:00.0: BAR 0 [mem 0xc0000000-0xc0ffffff 64bit]: releasing
[    5.465240] i915 0000:07:00.0: BAR 2 [mem 0x381000000000-0x38100fffffff 64bit pref]: releasing
[    5.465377] i915 0000:07:00.0: [drm] Failed to resize BAR2 to 8192M (-EINVAL)
[    5.465421] i915 0000:07:00.0: BAR 2 [mem 0x381000000000-0x38100fffffff 64bit pref]: assigned
[    5.465584] i915 0000:07:00.0: BAR 0 [mem 0xc0000000-0xc0ffffff 64bit]: assigned
[    5.482381] i915 0000:07:00.0: [drm] Using a reduced BAR size of 256MiB. Consider enabling 'Resizable BAR' or similar, if available in the BIOS.
[    5.500437] i915 0000:07:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[    5.505764] i915 0000:07:00.0: [drm] Finished loading DMC firmware i915/dg2_dmc_ver2_08.bin (v2.8)
[    5.689274] i915 0000:07:00.0: [drm] GT0: GuC firmware i915/dg2_guc_70.bin version 70.36.0
[    5.689280] i915 0000:07:00.0: [drm] GT0: HuC firmware i915/dg2_huc_gsc.bin version 7.10.16
[    5.700292] i915 0000:07:00.0: [drm] GT0: GUC: submission enabled
[    5.700299] i915 0000:07:00.0: [drm] GT0: GUC: SLPC enabled
[    5.700539] i915 0000:07:00.0: [drm] GT0: GUC: RC enabled
[    5.769601] [drm] Initialized i915 1.6.0 for 0000:07:00.0 on minor 1
[    5.796015] i915 0000:07:00.0: [drm] Cannot find any crtc or sizes
[    5.871997] i915 0000:07:00.0: [drm] Cannot find any crtc or sizes
[    5.896996] i915 0000:07:00.0: [drm] Cannot find any crtc or sizes
[    6.562092] i915 0000:07:00.0: [drm] GT0: HuC: authenticated for all workloads
[    6.562106] mei_pxp i915.mei-gsc.1792-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:07:00.0 (ops i915_pxp_tee_component_ops [i915])
[  403.021558] fbcon: i915drmfb (fb0) is primary device
[  403.078347] i915 0000:07:00.0: [drm] fb0: i915drmfb frame buffer device
PLEX

Code: Select all

root@plex:/home/zemerdon# apt install plexmediaserver
Installing:                     
  plexmediaserver

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 0
  Download size: 83.1 MB
  Space needed: 221 MB / 191 GB available

Get:1 https://downloads.plex.tv/repo/deb public/main amd64 plexmediaserver amd64 1.41.9.9961-46083195d [83.1 MB]
Fetched 83.1 MB in 3s (32.2 MB/s)          
Selecting previously unselected package plexmediaserver.
(Reading database ... 127583 files and directories currently installed.)
Preparing to unpack .../plexmediaserver_1.41.9.9961-46083195d_amd64.deb ...
PlexMediaServer install: Pre-installation Validation.
PlexMediaServer install: Pre-installation Validation complete.
Unpacking plexmediaserver (1.41.9.9961-46083195d) ...
Setting up plexmediaserver (1.41.9.9961-46083195d) ...
PlexMediaServer install: PlexMediaServer-1.41.9.9961-46083195d - Installation starting.
PlexMediaServer install: 
PlexMediaServer install: Now installing based on:
PlexMediaServer install:   Installation Type:   New
PlexMediaServer install:   Process Control:     systemd
PlexMediaServer install:   Plex User:           plex
PlexMediaServer install:   Plex Group:          plex
PlexMediaServer install:   Video Group:         render
PlexMediaServer install:   Metadata Dir:        /var/lib/plexmediaserver/Library/Application Support
PlexMediaServer install:   Temp Directory:      /tmp 
PlexMediaServer install:   Lang Encoding:       en_US.UTF-8
PlexMediaServer install:   Processor:           Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz
PlexMediaServer install:   Intel i915 Hardware: Found
PlexMediaServer install:   Nvidia GPU card:     Not Found
PlexMediaServer install:  
PlexMediaServer install: Completing final configuration.
Created symlink '/etc/systemd/system/multi-user.target.wants/plexmediaserver.service' → '/usr/lib/systemd/system/plexmediaserver.service'.
PlexMediaServer install: PlexMediaServer-1.41.9.9961-46083195d - Installation successful.  Errors: 0, Warnings: 0
Scanning processes...                                                                                                                             
Scanning linux images...                                                                                                                          

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@plex:/home/zemerdon# 
Tested Plex transcoding works like a dream.
Rebooted the VM via virt-manager and it came back to life, no issues, transcoding no issues as well. Marking this as working.
Post Reply