kvm 显卡直通(单显卡)

本文最后更新于:2022年8月9日 下午

1 安装

1
2
3
4
5
app-emulation/qemu
app-emulation/libvirt
app-emulation/virt-manager
net-misc/bridge-utils
sys-firmware/edk2-ovmf
1
2
3
4
usermod -a -G libvirt miloo
usermod -a -G kvm miloo
systemctl enable libvirtd
systemctl enable virtlogd-admin.socket

2 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
worisur ~ # neofetch
-/oyddmdhs+:. root@worisur
-odNMMMMMMMMNNmhy+-` ------------
-yNMMMMMMMMMMMNNNmmdhy+- OS: Gentoo Base System release 2.8 x86_64
`omMMMMMMMMMMMMNmdmmmmddhhy/` Kernel: 5.15.52-gentoo-dist
omMMMMMMMMMMMNhhyyyohmdddhhhdo` Uptime: 19 hours, 43 mins
.ydMMMMMMMMMMdhs++so/smdddhhhhdm+` Packages: 975 (emerge)
oyhdmNMMMMMMMNdyooydmddddhhhhyhNd. Shell: bash 5.1.16
:oyhhdNNMMMMMMMNNNmmdddhhhhhyymMh Terminal: /dev/pts/0
.:+sydNMMMMMNNNmmmdddhhhhhhmMmy CPU: AMD Ryzen 9 5900X (24) @ 3.700GHz
/mMMMMMMNNNmmmdddhhhhhmMNhs: GPU: AMD ATI Radeon RX 6800/6800 XT / 6900 XT
`oNMMMMMMMNNNmmmddddhhdmMNhs+` Memory: 33068MiB / 64220MiB
`sNMMMMMMMMNNNmmmdddddmNMmhs/.
/NMMMMMMMMNNNNmmmdddmNMNdso:`
+MMMMMMMNNNNNmmmmdmNMNdso/-
yMMNNNNNNNmmmmmNNMmhs+/-`
/hMMNNNNNNNNMNdhs++/-`
`/ohdmmddhys+++/:.`
`-//////:--.

2.1 桥接

无线网卡看这里

brctl配置看这里

  • 我的做法:

NetworkManager 直接使用 nmtui 将需要桥接的网卡加进新建的网桥(并将该网卡的mac地址配置给网桥),然后给网桥配置静态的地址(配置方式和普通网卡没区别),设置开机启动

2.2 iommu

1
2
3
4
5
6
cat /etc/default/grub
# 添加如下行
# AMD处理器(根据BIOS设置默认已启用?):amd_iommu=on
# 可视情况添加(修复或导致黑屏):iommu=pt
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt"

重启后,看看启动日志检查一下IOMMU是否已启用,使用超级用户权限运行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
worisur ~ # dmesg | grep -e DMAR -e iommu
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.15.52-gentoo-dist root=UUID=c9ffa84e-ef8f-45f0-b508-18d12102fbbc ro amd_iommu=on iommu=pt
[ 0.067146] Kernel command line: BOOT_IMAGE=/vmlinuz-5.15.52-gentoo-dist root=UUID=c9ffa84e-ef8f-45f0-b508-18d12102fbbc ro amd_iommu=on iommu=pt
[ 0.531545] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.547978] pci 0000:00:01.0: Adding to iommu group 0
[ 0.547984] pci 0000:00:01.1: Adding to iommu group 1
[ 0.547991] pci 0000:00:01.2: Adding to iommu group 2
[ 0.547997] pci 0000:00:02.0: Adding to iommu group 3
[ 0.548004] pci 0000:00:03.0: Adding to iommu group 4
[ 0.548010] pci 0000:00:03.1: Adding to iommu group 5
[ 0.548016] pci 0000:00:04.0: Adding to iommu group 6
[ 0.548022] pci 0000:00:05.0: Adding to iommu group 7
[ 0.548029] pci 0000:00:07.0: Adding to iommu group 8
[ 0.548034] pci 0000:00:07.1: Adding to iommu group 9
[ 0.548040] pci 0000:00:08.0: Adding to iommu group 10
[ 0.548046] pci 0000:00:08.1: Adding to iommu group 11
[ 0.548054] pci 0000:00:14.0: Adding to iommu group 12
[ 0.548059] pci 0000:00:14.3: Adding to iommu group 12
[ 0.548075] pci 0000:00:18.0: Adding to iommu group 13
[ 0.548079] pci 0000:00:18.1: Adding to iommu group 13
[ 0.548084] pci 0000:00:18.2: Adding to iommu group 13
[ 0.548089] pci 0000:00:18.3: Adding to iommu group 13
[ 0.548093] pci 0000:00:18.4: Adding to iommu group 13
[ 0.548098] pci 0000:00:18.5: Adding to iommu group 13
[ 0.548102] pci 0000:00:18.6: Adding to iommu group 13
[ 0.548107] pci 0000:00:18.7: Adding to iommu group 13
[ 0.548112] pci 0000:01:00.0: Adding to iommu group 14
[ 0.548122] pci 0000:02:00.0: Adding to iommu group 15
[ 0.548128] pci 0000:02:00.1: Adding to iommu group 15
[ 0.548133] pci 0000:02:00.2: Adding to iommu group 15
[ 0.548136] pci 0000:03:00.0: Adding to iommu group 15
[ 0.548139] pci 0000:03:04.0: Adding to iommu group 15
[ 0.548142] pci 0000:03:08.0: Adding to iommu group 15
[ 0.548144] pci 0000:03:09.0: Adding to iommu group 15
[ 0.548147] pci 0000:05:00.0: Adding to iommu group 15
[ 0.548150] pci 0000:06:00.0: Adding to iommu group 15
[ 0.548153] pci 0000:07:00.0: Adding to iommu group 15
[ 0.548159] pci 0000:08:00.0: Adding to iommu group 16
[ 0.548165] pci 0000:09:00.0: Adding to iommu group 17
[ 0.548289] pci 0000:0a:00.0: Adding to iommu group 18
[ 0.548297] pci 0000:0a:00.1: Adding to iommu group 19
[ 0.548306] pci 0000:0a:00.2: Adding to iommu group 20
[ 0.548314] pci 0000:0a:00.3: Adding to iommu group 21
[ 0.548320] pci 0000:0b:00.0: Adding to iommu group 22
[ 0.548326] pci 0000:0c:00.0: Adding to iommu group 23
[ 0.548332] pci 0000:0c:00.1: Adding to iommu group 24
[ 0.548338] pci 0000:0c:00.3: Adding to iommu group 25
[ 0.548344] pci 0000:0c:00.4: Adding to iommu group 26
[ 0.549234] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

查看显卡相关信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;

# 结果

IOMMU Group 16:
08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 17:
09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 18:
0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 19:
0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 20:
0a:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 21:
0a:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]

2.3 内核模块添加

1
2
3
4
5
# 配置ids内容
worisur ~ # cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4,1002:1478,1002:1479
options vfio-pci disable_idle_d3=1
options vfio-pci disable_vga=1

2.4 hook 脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
worisur /etc/libvirt/hooks # cat qemu
#!/bin/bash
OBJECT="$1"
OPERATION="$2"

if [ $OBJECT == "miloo" ] || [ $OBJECT == "mod" ] || [ $OBJECT == "work" ]; then
case "$OPERATION" in
"prepare")
systemctl start libvirt-nosleep@"$OBJECT" 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
/bin/vfio-startup.sh 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
;;

"release")
systemctl stop libvirt-nosleep@"$OBJECT" 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
/bin/vfio-teardown.sh 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
;;
esac
fi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
worisur /etc/libvirt/hooks # cat vfio-startup.sh
#!/bin/bash
# Helpful to read output when debugging
set -x

long_delay=10
medium_delay=5
short_delay=1
echo "Beginning of startup!"

function stop_display_manager_if_running {
# Stop dm using systemd
if command -v systemctl; then
if systemctl is-active --quiet "$1.service" ; then
echo $1 >> /tmp/vfio-store-display-manager
systemctl stop "$1.service"
fi

while systemctl is-active --quiet "$1.service" ; do
sleep "${medium_delay}"
done

return
fi

# Stop dm using runit
if command -v sv; then
if sv status $1 ; then
echo $1 >> /tmp/vfio-store-display-manager
sv stop $1
fi
fi
}

# Stop currently running display manager
if test -e "/tmp/vfio-store-display-manager" ; then
rm -f /tmp/vfio-store-display-manager
fi

stop_display_manager_if_running gdm
stop_display_manager_if_running display-manager

sleep "${medium_delay}"

# Unbind VTconsoles if currently bound (adapted from https://www.kernel.org/doc/Documentation/fb/fbcon.txt)
if test -e "/tmp/vfio-bound-consoles" ; then
rm -f /tmp/vfio-bound-consoles
fi
for (( i = 0; i < 16; i++))
do
if test -x /sys/class/vtconsole/vtcon${i}; then
if [ `cat /sys/class/vtconsole/vtcon${i}/name | grep -c "frame buffer"` \
= 1 ]; then
echo 0 > /sys/class/vtconsole/vtcon${i}/bind
echo "Unbinding console ${i}"
echo $i >> /tmp/vfio-bound-consoles
fi
fi
done

# Unbind EFI-Framebuffer
if test -e "/tmp/vfio-is-nvidia" ; then
rm -f /tmp/vfio-is-nvidia
fi

if lsmod | grep "nvidia" &> /dev/null ; then
echo "true" >> /tmp/vfio-is-nvidia
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
fi

echo "End of startup!"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
worisur /etc/libvirt/hooks # cat vfio-teardown.sh
#!/bin/bash
set -x

echo "Beginning of teardown!"

sleep 10

# Restart Display Manager
input="/tmp/vfio-store-display-manager"
while read displayManager; do
if command -v systemctl; then
systemctl start "$displayManager.service"
else
if command -v sv; then
sv start $displayManager
fi
fi
done < "$input"

# Rebind VT consoles (adapted from https://www.kernel.org/doc/Documentation/fb/fbcon.txt)
input="/tmp/vfio-bound-consoles"
while read consoleNumber; do
if test -x /sys/class/vtconsole/vtcon${consoleNumber}; then
if [ `cat /sys/class/vtconsole/vtcon${consoleNumber}/name | grep -c "frame buffer"` \
= 1 ]; then
echo "Rebinding console ${consoleNumber}"
echo 1 > /sys/class/vtconsole/vtcon${consoleNumber}/bind
fi
fi
done < "$input"

# Rebind framebuffer for nvidia
if test -e "/tmp/vfio-is-nvidia" ; then
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind
fi

echo "End of teardown!"
1
2
3
4
5
6
7
8
# 权限为 kvm 用户 组为 libvirt
worisur /etc/libvirt/hooks # ls -al
总用量 20
drwxr-xr-x 2 miloo libvirt 4096 7月 14 23:35 .
drwxr-xr-x 7 root root 4096 7月 13 01:25 ..
-rwxr-xr-x 1 miloo libvirt 622 7月 14 23:32 qemu
-rwxr-xr-x 1 miloo libvirt 1981 7月 13 02:06 vfio-startup.sh
-rwxr-xr-x 1 miloo libvirt 972 7月 13 02:07 vfio-teardown.sh

2.5 QEMU 配置

1
2
3
4
5
6
7
8
9
10
11
# /etc/libvirt/libvirtd.conf 修改如下行:
worisur /etc/libvirt # cat libvirtd.conf
unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"
log_filters="1:qemu"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"

# # /etc/libvirt/qemu.conf 修改如下行:
worisur /etc/libvirt # cat qemu.conf
user = "miloo"
group = "libvirt"

3 直通

3.1 创建不直通的虚拟机

  • 下载驱动
  • 芯片组:Q35
  • 固件:ovmf_code.fd

3.2 vbios

gpuz提取

此处下载

3.3 添加直通设备

  1. 打开虚拟机管理器,把显卡的所有pci部分都添加到虚拟机里面。

    pci设备:1002:73bf,1002:ab28,1002:73a6,1002:73a4,1002:1478,1002:1479

  2. 启用XML编辑, 在显卡的所有pci设备里面, 在之后添加: <rom bar="on" file="vbios路径"/>
  3. 把串口,显示协议,信道,usb转接,显卡qxl,触摸板那些东西删掉
  4. 添加你的USB设备,比如鼠标、键盘和USB耳机。

4 问题解决

4.1 软件不能运行在虚拟机上(Sorry, this program cannot run under a virtual machine)

  • Pass the SMBIOS information to the guest
  • Pass the exact host’s CPU model to the guest
  • Hide the “hypervisor” feature from the CPU
  • In the “features > hyperv” section, add a vendor_id
  • In the “features” section, hide kvm.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<os>
<smbios mode='host'/>
...
</os>
...
<features>
<hyperv>
...
<vendor_id state='on' value='123456789ab'/>
</hyperv>
<kvm>
<hidden state='on'/>
<hint-dedicated state='on'/>
</kvm>
...
</features>
...
<cpu mode='host-passthrough'>
...
<feature policy='disable' name='hypervisor'/>
</cpu>

4.2 黑屏(ucsi_cgg 1-0008: ucsi_handle_connector_change: GET_CONNECTOR_STATUS failed)

  • ucsi_cgg 为英伟达显卡的 mod 直接屏蔽就好
1
2
/etc/modprobe.d # cat blacklist-nvidia-usb.conf
blacklist ucsi_ccg

4.2 docker 桥接占用物理网卡导致网络不通的情况

  • 添加或修改 /etc/docker/daemon.json 将 kvm 的桥接网卡直接赋给 docker
1
2
3
4
{
"bridge": "virbr",
"default-gateway": "*.*.*.*"
}

kvm 显卡直通(单显卡)
https://worisur.github.io/2022/07/14/2022-07-14.kvm单显卡直通/
作者
worisur
发布于
2022年7月14日
许可协议