Proxmox 7.4 使用vgpu_unlock，为GTX1060开启vGPU支持

本文介绍: 本文实现通过4个脚本，实现利用消费级显卡实现vGPU。包含了对 Proxmox VE的更新，启用iommu，安装 nvidia grid 驱动，以及加载 vgpu_unlock。

本文在 2021年发布的博客《Proxmox 5.4使用vgpu_unlock，为GTX1060开启vGPU支持》，介绍了 Proxmox VE 5.4 上部署vGPU unlock 的操作步骤。

后续有发布了在 Proxmox VE 7.x 上支持 vGPU的博客《Proxmox 7.2 部署 DoraCloud桌面云，支持vGPU》，实现了通过 3个脚本完成 vGPU的配置。

本文在上述两个文章的基础上，实现通过4个脚本，实现利用消费级显卡实现vGPU。包含了对 Proxmox VE的更新，启用iommu，安装 nvidia grid 驱动，以及加载 vgpu_unlock。

准备工作：

硬件：一台服务器，配置有ssd硬盘，安装有GTX 1060 显卡。

软件：安装了 Proxmox VE 7.4 。

网络：网络中可以访问互联网。

步骤1：换源，更新

curl -o- http://www1.deskpool.com:9000/software/gpu01.sh |bash

上面的脚本的内容，为了方便阅读，贴在这里

#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# DoraCloud for Proxmox Enable vGPU
# Phase 1: update source to mirrors.ustc.edu.cn

cp /etc/apt/sources.list /etc/apt/sources.list.backup
sed -i 's|^deb http://ftp.debian.org|deb https://mirrors.ustc.edu.cn|g' /etc/apt/sources.list
sed -i 's|^deb http://security.debian.org|deb https://mirrors.ustc.edu.cn/debian-security|g' /etc/apt/sources.list

mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
CODENAME=`cat /etc/os-release |grep PRETTY_NAME |cut -f 2 -d "(" |cut -f 1 -d ")"`

#echo "deb https://mirrors.nju.edu.cn/proxmox/debian/pve $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
#echo "deb https://mirrors.ustc.edu.cn/proxmox/debian $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list

wget https://mirrors.ustc.edu.cn/proxmox/debian/proxmox-release-$CODENAME.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-$CODENAME.gpg

#更新
apt update && apt install  pve-kernel-5.15 pve-headers-5.15 dkms mdevctl -y

步骤2：启用iommu

curl -o- http://www1.deskpool.com:9000/software/gpu02.sh |bash

上面的脚本的内容，为了方便阅读，贴在下方

#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Enable IO-MMU on PVE Server


# 复制如下脚本，启用IO-MMU
echo ""
echo "********************************************"
echo "***  Enable IO-MMU on proxmox host       ***"
echo "********************************************"
# /etc/default/grub 的GRUB_CMDLINE_LINUX_DEFAULT，增加 intel_iommu=on iommu=pt
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"/g' /etc/default/grub

echo ""
echo "    Update grub .... "
update-grub

# 加载 vfio vfio_iommu_type1 vfio_pci vfio_virqfd 4个Modules
echo vfio >> /etc/modules
echo vfio_iommu_type1 >> /etc/modules
echo vfio_pci >> /etc/modules
echo vfio_virqfd >> /etc/modules


echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
echo "blacklist nouveau" >>/etc/modprobe.d/disable-nouveau.conf
echo "options nouveau modeset=0" >>/etc/modprobe.d/disable-nouveau.conf

update-initramfs -u
echo ""
echo "    Proxmox host reboot ............."
reboot

步骤3：安装 NVIDIA GRID 13.6的驱动

curl -o- http://www1.deskpool.com:9000/software/gpu03e.sh |bash

上面的脚本的内容，请自己阅读。这里选择相对稳定的 13.6 版本。

#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Install NVIDIA Linux vGPU Driver 470.161.02
echo ""
echo "********************************************************"
echo "*** Install NVIDIA Grid vGPU Driver 13.6 V470.161.02 ***"
echo "********************************************************"

echo "    Downloading NVIDIA driver "
wget http://www1.deskpool.com:9000/software/NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run
chmod +x NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run

echo ""
echo "    Extracting driver .... "
./NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run -x

cd NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm/

echo ""
echo "    Patching NVIDIA driver "
sed -i 's|.open             = nv_vgpu_vfio_open|.open_device = nv_vgpu_vfio_open|g'    kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c
sed -i 's|.release          = nv_vgpu_vfio_close|.close_device = nv_vgpu_vfio_close|g'    kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c

echo ""
echo "    Installing driver .... "
./nvidia-installer -dkms -s

echo ""
echo "    deamon-reload && rebooting ...."
systemctl daemon-reload
reboot

步骤4：运行 vgpu_unlock 和 vgpu_unlock-rs

curl -o- http://www1.deskpool.com:9000/software/gpu03u.sh |bash

上面的脚本的内容，请自己阅读。

#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Unlock NVIDIA GTX 1060

echo ""
echo "********************************************************"
echo "***  Unlock NVIDIA GTX1060                           ***"
echo "********************************************************"


echo ""
echo ""
echo "    Download vgpu_lock "
echo ""
apt install git-core -y

git clone https://gitee.com/deskpool/vgpu_unlock.git
chmod -R +x vgpu_unlock


echo ""
echo ""
echo "********************************************************"
echo "    Apply vgpu_unlock to 470.161.02"
echo ""
sed -i 's/#include "nv-time.h"/#include "nv-time.h"nn#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"/g' /usr/src/nvidia-470.161.02/nvidia/os-interface.c
echo "ldflags-y += -T /root/vgpu_unlock/kern.ld" >> /usr/src/nvidia-470.161.02/nvidia/nvidia.Kbuild
dkms remove -m nvidia -v 470.161.02
dkms install -m nvidia -v 470.161.02




echo ""
echo ""
echo "********************************************************"
echo "    Install RUST "
echo ""
#设置rustup的source为 USTC
export RUSTUP_DIST_SERVER=https://mirrors.ustc.edu.cn/rust-static
export RUSTUP_UPDATE_ROOT=https://mirrors.ustc.edu.cn/rust-static/rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
#wget -qO- https://cdn.jsdelivr.net/gh/rust-lang-nursery/rustup.rs/rustup-init.sh |sh  -s -- -y

echo ""
echo ""
echo "********************************************************"
echo "    Config cargo "
echo ""
#加载cargo的环境变量
source ~/.cargo/env

#设置cargo的源镜像为 USTC
cat >~/.cargo/config <<EOF
[source.crates-io]
replace-with = 'ustc'

[source.ustc]
registry = "git://mirrors.ustc.edu.cn/crates.io-index"
EOF


echo ""
echo ""
echo "********************************************************"
echo "   Building vgpu_unlock-rs  "
echo ""

#下载 vgpu_unlock-rs 项目
git clone https://gitee.com/deskpool/vgpu_unlock-rs
cd vgpu_unlock-rs/
cargo build --release

echo ""
echo ""
echo "********************************************************"
echo "   Create vgpu.service && vgpu-mgr.service "
echo ""

#nvidia-vgpud 的unlock 服务
mkdir /etc/systemd/system/nvidia-vgpud.service.d
cat >/etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf <<EOF
[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so
EOF

#nvidia-vgpu-mgr 的unlock服务
mkdir /etc/systemd/system/nvidia-vgpu-mgr.service.d
cat >/etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf <<EOF
[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so
EOF

#描述文件，控制GPU的显示配置，以及cuda
mkdir /etc/vgpu_unlock
cat >/etc/vgpu_unlock/profile_override.toml <<EOF
[profile.nvidia-55]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
EOF

echo ""
echo ""
echo "********************************************************"
echo "    Reboot Proxmox VE .....  "
echo "    run  mdevctl types to check vGPU is installed.     "
echo ""

sync
reboot

里面的内容包括了安装 vgpu_unlock ，安装 rust 环境，下载和编译 vgpu_unlock-rs，配置 nvidia-vgpu 和nvidia-vgpu-mgr的服务，

生成vgpu_unlock 显卡配置文件 profile_override.toml 等内容。

全部执行完毕后，系统重启之后，运行 mdevctl types ，可以检查到 vGPU的类型已经生成。

root@pve03:~# mdevctl types
0000:01:00.0
  nvidia-156
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-215
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-241
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-283
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4C
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
  nvidia-284
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6C
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
  nvidia-285
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8C
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
  nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
  nvidia-287
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24C
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
  nvidia-46
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-47
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-48
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-49
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-50
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-51
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-52
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-53
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-54
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-55
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-56
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-57
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-58
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-59
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-60
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-61
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
  nvidia-62
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24

剩下的工作是部署 GuestOS，并安装 NVIDIA GRID 驱动。这里推荐部署 DoraCloud，下载 win10LTSC2021V3 模板。内置了GRID 13.6的驱动。

部署的方法，参考《Proxmox 7.2 部署 DoraCloud桌面云，支持vGPU》，以及《ShareStation工作站虚拟化部署 (doracloud.cn) 》

最后分享以下， GTX 1060 虚拟化之后，vGPU 运行《原神》游戏的效果。