PCI Passthrough From Start- Advice Needed

General support questions
Post Reply
SteveFromAfrica
Posts: 1
Joined: 2019/06/14 18:16:36

PCI Passthrough From Start- Advice Needed

Post by SteveFromAfrica » 2019/06/14 18:24:45

Good day, I am starting a pci passthrough build. I could use a bit of handholding.
I get the following errors code 31 , 43

setup below, any assistance would be most welcome

Server
Media Install process CentOS
1) set time zone
2) setup network card to on also setup manual 10,0,05
3) make note of WAN Card (eno5) Mac 0C:C4:7A:9F:27:8C
4) set user and root
user = steven (root same)
Post Install
1) yum -y update as root / su
2)SSH -V
systemctl restart sshd.service (remote access from client)
3) rename server hostnamectl set-hostname Skippy
hostnamectl
4) yum install vim-X11

KVM Setup
1) yum install virt-install qemu-kvm libvirt libvirt-python libguestfs-tools virt-manager -y
Enable and Start the Services
# systemctl enable libvirtd
# systemctl start libvirtd
# systemctl status libvirtd
2) modinfo kvm to check

KVM Storage Pools
1) check discks run lsblk
2) # vgcreate KVM_Storage /dev/sdb1
# lvcreate -l +100%FREE -n KVM_lv KVM_Storage
#mkfs.xfs /dev/mapper/KVM_Storage_KVM_lv
3) edit /etc/fstab with vi

## KVM VM Storage
/dev/mapper/KVM_Storage-KVM_lv /var/lib/libvirt/images xfs defaults 0 0
4) df -h This will show current
5) mount -a
Virsh
1) virsh pool-list
2) virsh pool-define-as KVM_StoragePool --type dir --target /var/lib/libvirt/images
4) virsh pool-start KVM_StoragePool
6) virsh pool-autostart KVM_StoragePool
ovmf centos
1) wget -O /etc/yum.repos.d/firmware.repo
2) yum -y install edk2.git-ovmf-x64
iOMMU GPU Configure
1) vi /etc/default/grub
2) grub2-mkconfig -o /boot/grub2/grub.cfg
3) reboot
4) dmesg | grep -E "DMAR|IOMMU”
5) [root@skippy ~]# dmesg | grep -E "DMAR|IOMMU"
[ 0.000000] ACPI: DMAR 0000000079957f50 000CC (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[ 0.000000] DMAR: IOMMU enabled
[ 0.062950] DMAR: Host address width 46
[ 0.062953] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[ 0.062960] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[ 0.062962] DMAR: RMRR base: 0x0000007bb24000 end: 0x0000007bb32fff
[ 0.062964] DMAR: ATSR flags: 0x0
[ 0.062965] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 0.062968] DMAR-IR: IOAPIC id 8 under DRHD base 0xfbffc000 IOMMU 0
[ 0.062969] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[ 0.062971] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.062972] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.063148] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 3.783426] DMAR: dmar0: Using Queued invalidation
[ 3.783444] DMAR: Setting RMRR:
[ 3.783508] DMAR: Setting identity map for device 0000:00:14.0 [0x7bb24000 - 0x7bb32fff]
[ 3.783603] DMAR: Setting identity map for device 0000:00:1a.0 [0x7bb24000 - 0x7bb32fff]
[ 3.783679] DMAR: Setting identity map for device 0000:00:1d.0 [0x7bb24000 - 0x7bb32fff]
[ 3.783701] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 3.783751] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 3.783766] DMAR: Intel(R) Virtualization Technology for Directed I/O

6) uname -a
Linux skippy 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

7) lspci -nn | grep -i nvidia
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P620] [10de:1cb6] (rev a1)
07:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
8) vi /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1cb6,10de:0fb9
9) echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
10) reboot
11) dmesg | grep -i vfio
[ 6.603241] VFIO - User Level meta-driver version: 0.3
[ 6.615629] vfio_pci: add [10de:1cb6[ffff:ffff]] class 0x000000/00000000
[ 6.615937] vfio_pci: add [10de:0fb9[ffff:ffff]] class 0x000000/00000000
12) /usr/libexec/qemu-kvm -version

13) yum -y install centos-release-qemu-ev
14) sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
15) yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
16) systemctl restart libvirtd
17) /usr/libexec/qemu-kvm -version 
18) /usr/libexec/qemu-kvm -machine help 

Supported machines are:
pc RHEL 7.4.0 PC (i440FX + PIIX, 1996) (alias of pc-i440fx-rhel7.4.0)
pc-i440fx-rhel7.4.0 RHEL 7.4.0 PC (i440FX + PIIX, 1996) (default)
pc-i440fx-rhel7.3.0 RHEL 7.3.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.2.0 RHEL 7.2.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.1.0 RHEL 7.1.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.0.0 RHEL 7.0.0 PC (i440FX + PIIX, 1996)
rhel6.6.0 RHEL 6.6.0 PC
rhel6.5.0 RHEL 6.5.0 PC
rhel6.4.0 RHEL 6.4.0 PC
rhel6.3.0 RHEL 6.3.0 PC
rhel6.2.0 RHEL 6.2.0 PC
rhel6.1.0 RHEL 6.1.0 PC
rhel6.0.0 RHEL 6.0.0 PC
q35 RHEL-7.4.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel7.4.0)
pc-q35-rhel7.4.0 RHEL-7.4.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.3.0 RHEL-7.3.0 PC (Q35 + ICH9, 2009)
none empty machine
Custom VM Settings in KVM Then Edit VM in virsh

Nvidia error 43 have to hide the id of the VM

1a) sudo virsh
1b) sudo EDITOR=vim virsh edit win10
2) <vendor_id state='on' value='123456789ab'/>
3) <kvm>
<hidden state='on'/>
</kvm>

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: PCI Passthrough From Start- Advice Needed

Post by hunter86_bg » 2019/06/15 18:44:49

KVM exposes the information to the OS that this is a VM.
If you use an end-user Nvidia GPU, the drivers in windows (based on other user experience) will prevent you to use your IOMMU Nvidia.
This is not valid for enterprise-grade GPUs. They work fine.

You have several options:
1. Use AMD
2. Use another distro which will not expose the virtio flags by default to Windows (or any other OS)
In windows , you can check if the KVM exposes the info in Windows Task Manager.
3. Hide from the OS the fact that it is actually a VM (somehow) and share that knowledge with us :)

You can always try the '-hypervisor' flag as described in:
https://www.google.com/amp/s/amp.reddit ... _detected/

Post Reply