NVIDIA's vGPU stack, running on a consumer RTX 3090 through the legacy mdev path, with guests that do real work: CUDA, compiled PTX kernels, OpenGL, NVENC. The host keeps the card at the same time and can run its own CUDA.
Start here: docs/TUTORIAL.md - the full walkthrough, bare machine to a working guest, with the reasoning and the failure mode at each step. It opens with five questions that tell you in two minutes whether this can work for you at all.
Read docs/HARDWARE-SCOPE.md before you start. This is proven on exactly one GPU — a GA102 RTX 3090 — and on exactly two kernels. It is not a general recipe and it will not work unmodified on your card.
If you cannot find something internet archive is a very good place to search for drivers.
What has been verified by reading data back, not by the absence of errors:
| result | |
|---|---|
| Guest CUDA | cuInit 0, device reports NVIDIA RTXA5000-8Q, compute capability 8.6, CUDA 12.2 |
| Guest compute | compiled PTX kernels run; a 4M-element fill verified element by element |
| Guest copy | 4 MiB round trip verified; ~11.1 GiB/s host-to-device, ~10.0 GiB/s back |
| Guest graphics | headless EGL/GLES renders and reads back correct pixels, all 6 FBO formats complete |
| Host CUDA | runs concurrently with a guest on the same card |
| Two guests | both held a slice and ran a 9B-parameter model, ~83 tok/s aggregate, time-sliced |
| Stability | 34609 iterations over 300 s with 0 failures; 200-iteration soak clean; no Xid |
The throughput and endurance figures above come from the project's own logs and were not re-run when this repository was assembled. The configuration facts in docs/KERNEL-REQUIREMENTS.md were measured directly off the running machine and are marked where they were not.
- Driver 535.309.01, a merged vgpu-kvm build. The RM core is unpatched.
- Kernel 6.1.71. Not 6.8, not 6.18. This is the part people get wrong; see below.
RMSetSriovMode=0, because consumer Ampere has no SR-IOV and legacy mdev is the only path.- The device-ID spoof is userspace only.
nvidia.koalways sees the real10de:2204. - One kprobe is mandatory. Without it the guest's graphics channel is scheduled once and then never again, with no error logged anywhere.
Same driver, same configuration, same machine. Only the kernel changed:
| kernel | outcome |
|---|---|
| 5.15.95 | works (recorded on a separate install; not re-measured, and see the note below) |
| 6.1.71 | works — this is what runs |
| 6.8.9 | fails: Immediate pteblit ... timed out, 50 timeouts, 100 plugin errors |
| 6.18.52 | fails the same way, worse: roughly 76 timeouts and 152 errors |
Nothing between 6.1 and 6.8 was tested, so the real boundary is somewhere in that gap and this project does not know where. The failure is quiet and slow: the stack loads, the mdev appears, the guest boots to a desktop, and then the framebuffer blit times out thousands of times. If you are on an untested kernel you will not get an obvious error, you will get something that looks almost right.
scripts/preflight.sh warns when the running kernel is not on the known-good list. Full detail,
including what was ruled out as the cause, is in docs/KERNEL-REQUIREMENTS.md.
docs/TUTORIAL.md the full walkthrough - read this first
docs/BUILDING-ON-DEBIAN.md Debian and Ubuntu: split headers, and the board mapping
docs/PROXMOX.md Proxmox on a kernel that works, and pinning it there
docs/PROXMOX-VM-USAGE.md running the guest as a Proxmox VM, adding more, snapshots,
and the teardown wedge that makes a stopped guest unstartable
docs/LICENSING.md self-hosted FastAPI-DLS, and the clock skew that silently breaks it
vgpu.conf.example every tunable, with defaults and auto-detection
lib/common.sh config loading, hardware discovery, sanity checks
scripts/vgpu one command for the whole rig: status/start/stop/fix/new/save/ip/log
scripts/preflight.sh report what this machine looks like; changes nothing
scripts/bringup.sh load the stack in the order that works
scripts/startguest.sh create an mdev and boot a guest on it
scripts/hardreset.sh full teardown, including the vfio core
scripts/vgpu-recover.sh clear the teardown wedge: stop all, reload the stack, start again
scripts/vgpu-provision.sh install driver, agent, Sunshine, display driver and licence
token into a guest; you supply the driver, it does the rest.
WRITTEN BUT NEVER RUN END TO END - read it before trusting it
scripts/vgpu-stress.sh regression harness: recover cycles asserting licence, Xid and
pte-blit all stay clean. Has completed single cycles, not yet a
clean multi-cycle run
scripts/vgpu-guest-timesync.sh keep guest clocks matched to the host - licensing breaks
silently when they drift
scripts/gexec.sh run a PowerShell snippet in a guest through the QEMU guest agent
scripts/pve-newvm.sh create another vGPU guest end to end, including the throwaway boot
scripts/pve-fix-rdp-wddm.sh set/unset fEnableWddmDriver offline (-r reverts); did NOT fix
the RDP hang here - kept because it is a verified way to toggle it
scripts/pve-fix-nvidia-regkeys.sh write the guest tuning keys into a stopped guest's
registry offline, for when Windows allocates a new class subkey
scripts/as-run/ the original machine-specific versions, kept for provenance
kmod/zfmulti/ required: suppresses the leaked runlist disable
kmod/mdguest/ loaded by the working setup; measurably inert on this build
kmod/isrfind/ finds the one-way call site on a driver build you do not have
an offset for yet - this is what makes other versions tractable
kmod/* read-only diagnostic probes used during the investigation
notes/ what was tried and what failed
COMPATIBILITY.md what works, what fails, what nobody has tried - please add rows
CONTRIBUTING.md what is most wanted, and the traps this codebase hit
Makefile make modules / make check / make install
Start with preflight.sh. It discovers the GPU, its IOMMU group, the module set for your
kernel, your OVMF path and the profiles your card offers, then tells you what is missing. It
touches nothing.
No monitor output. The vgpu-kvm RM core registers zero KMS connectors, so neither the host
nor a guest can drive a display from this card. Measured: /sys/class/drm contains nothing but
version, and /dev/dri does not exist. If you want a screen, put a second GPU in the machine.
Guests are reached over RDP, or by streaming.
No VRAM overcommit. Profiles carve the 24 GB into fixed slices. The 8Q profile used here is 8192 MB with a maximum of 3 concurrent instances.
Finding the drivers. Both the host and guest drivers come from NVIDIA's licensing portal and are not public downloads. If a version has been withdrawn or you cannot reach the portal, the Internet Archive is a genuinely good place to look - older vGPU and GRID installers are often mirrored there. Check the version matches what you need exactly, because the kprobe offset is tied to one driver build.
No restarting one guest on its own. Once any guest has been stopped, none will start again
until the NVIDIA kernel modules are reloaded — and that is true with nothing else running, so it
is not a contention problem. In practice the guests are one set that starts and stops together;
vgpu fix does the reload and brings them all back. Rebooting from inside Windows avoids it
entirely. Unsolved, with the eliminated causes recorded in
notes/WHAT-DIDNT-WORK.md.
Licensing is separate, and it bites hard. Unlocking the device gating has nothing to do with
vGPU licensing. Once the grace period expires a guest reports Unlicensed (Restricted) and the
GPU is throttled — measured here as a game dropping from 60 fps to 15 with the host GPU at 1–7 %
utilisation and no error anywhere. It reads exactly like a driver fault and is not one. Check
nvidia-smi -q | grep 'License Status' in the guest first when performance collapses for no
visible reason. docs/LICENSING.md covers running a self-hosted licence server
and the guest clock skew that silently prevents it from issuing a lease.
Not a supported configuration. Every driver and kernel update can break it, and the kprobe is tied to one driver build. Expect to re-derive things.
This builds directly on prior work: the unlock technique from vgpu_unlock (DualCoder), the
vgpu_unlock-rs library (mbilker) that actually runs here, and the vGPU-Unlock-Patcher
(VGPU-Community-Drivers) general-merge target that produced the driver. NVIDIA's open GPU
kernel modules source was the reference that made the closed host blob readable.
Full list, including the tooling and what was consulted and rejected: REFERENCES.md.
No NVIDIA code, driver installer or license is redistributed in this repository.
Be clear about this before trusting the code.
Verified by running it on the hardware:
-
scripts/preflight.sh- run against the live host. It correctly discovers the GPU, its real PCI id, the IOMMU group, the module set and vermagic, the OVMF path, all 18 profiles and the zero DRM connectors, and reports OK. -
The backing-file guard in
lib/common.sh- run against the real 4-deep qcow2 chain. It permits the leaf and refuses all three parents, each naming the overlay that blocks it. -
kmod/zfmulti,kmod/mdguest,kmod/isrfind- all build warning-free against kernel 6.1.71. -
make modulesandmake check- run end to end on the host. -
Everything in
scripts/as-run/- that is the exact text that produced every measurement quoted in the documentation. -
The full cycle, on a second machine (Debian 12 / Proxmox, kernel 6.1.0-53), from a fresh
git clonerather than the working tree:hardreset.sh→bringup.sh→startguest.sh→ clean guest shutdown.hardresetdiscovered the GPU and IOMMU group, unloaded everything on the first attempt and confirmed/dev/vfio/15destroyed.bringupreturned 0 with 18 profiles,nvidia-664 avail=3,xid 0.startguestresolved the profile by name, created the mdev, bound VNC to loopback and had RDP up in 40 seconds. On shutdown the wrapper released the mdev —available_instanceswent back to 3 — which is the behaviour that matters, because QEMU aborts during vfio teardown and a hand-rolled launcher leaks an mdev every run.
What that cycle found, which reading the code had not:
Three defects, all in scripts that passed bash -n and shellcheck -S warning:
bringup.shcarried a two-character escape where a line continuation was meant, sosystemctl is-activereceived an extra argument and the script died on every run.bound_driver()usedreadlink -fon the PCIdriversymlink. That call succeeds when the final component is missing and returns the path itself, so with no driver bound it reported the literal stringdriverandbringup.shrefused to reset a free device. The identical bug had already been found and fixed ingpu_iommu_group, with a comment explaining it, and was not carried across to the function added later in the same file.- The
nvidia-vgpudcheck usedsystemctl is-active. That unit isType=oneshot, so it reportsinactiveafter a successful run — the check failed every time it worked. The oneshot behaviour was already written down in two documents in this repository.
The pattern is worth stating plainly: every one of those was a fact the project already knew, written down, and then not applied to code added in the same pass. Static checking caught none of them. Running the thing caught all three.
If you want the proven-exact path, scripts/as-run/ is still the text that produced the original
measurements.
These are the exact failures this repository explains, with the symptom first. If you are staring at one of them right now, the linked section is where the answer is.
| what you are seeing | what it means |
|---|---|
Immediate pteblit ... timed out, hundreds of them, from nvidia-vgpu-mgr |
your kernel. 6.8 and 6.18 fail this way, 6.1.71 does not. KERNEL-REQUIREMENTS.md |
| guest shows Code 43 in Device Manager with a vGPU attached | the two guest registry keys are missing, or are in the wrong class subkey. Tutorial Part 9. If the guest only started doing this after you moved it to Proxmox or changed its PCI slot, the keys are on the old subkey: PROXMOX-VM-USAGE.md |
Xid 44 ... Ch 00000008, intr 00000000 every run |
RMSetClientRMAllocatedCtxBuffer is unset, so the guest promotes its own context buffers. Tutorial Part 9 |
| guest boots, GPU present, no errors anywhere, and nothing renders | an interrupt handler disabled the graphics runlist and nothing re-enabled it. This is the core finding. Tutorial Part 5 |
init_device_instance fails with error 7 |
more than one cause. After a by-hand driver reload, no PCI function-level reset. After a guest has been stopped, the teardown wedge — see the next row |
Timed out (6001 ms) trying to sync / init_device_instance failed ... (init frame copy engine) / start failed. status: 0x1 |
the teardown wedge: once any guest has been stopped, none will start until the modules are reloaded. vgpu fix. Unsolved; the eliminated causes are listed in PROXMOX-VM-USAGE.md |
Unlicensed (Restricted) in nvidia-smi -q, and everything is suddenly slow |
the licence grace period expired and the GPU is throttled. LICENSING.md |
licence client loops on POST /auth/v1/origin and never gets a lease |
the guest clock disagrees with the licence server; the tokens are time-signed. LICENSING.md |
Failed to find the hardcoded NLS certificates! from gridd-unlock-patcher |
the patcher does not know this driver build — and on 539.72 it is not needed at all. LICENSING.md |
Xid 31 ... MMU Fault: ENGINE NVENC0 |
the video encoder faulting on HEVC yuv444 10-bit. Constrain the encoder to H.264 (hevc_mode = 1, av1_mode = 1) |
NvEnc: encode wait timeout repeating, with the stream hitching |
something in the guest is hammering the GPU's control path. FurMark's GUI reproduces it; ordinary games do not |
snapshot feature is not available |
efidisk0 is raw, which cannot snapshot. Convert it to qcow2. PROXMOX-VM-USAGE.md |
insmod: Unknown symbol in module on nvidia-vgpu-vfio.ko |
mdev, vfio, vfio_pci_core or irqbypass is not loaded |
error getting device from group N ... not already in use |
a SIGKILLed QEMU wedged the IOMMU group; the vfio core modules must come out |
cuCtxCreate returns 801 NOT_SUPPORTED |
you are on an A profile. Use a Q profile. |
Xid 119 GSP_INIT_DONE timeout, mdev types vanish |
GSP cannot boot on this card. NVreg_EnableGpuFirmware=0 |
nvidia-smi says Virtualization Mode : None while a guest is running |
expected here, and not a useful signal. Check /sys/bus/mdev/devices/ |
krcWatchdogInit_IMPL returns NV_ERR_INVALID_STATE |
RmRcWatchdog=0 is missing in the guest |
no KMS connectors, /dev/dri absent, no monitor output |
structural. The vgpu-kvm RM core has no display engine. HARDWARE-SCOPE.md |
For the symbol _nv042311rm and the offset +0x28: that is the interrupt handler branch this
project suppresses, and FINDING-THE-ISR-OFFSET.md explains how to
locate the equivalent on a driver build other than 535.309.01.
This is a GeForce RTX 3090 running NVIDIA vGPU through the legacy mdev path - the same
territory as vgpu_unlock, but the problem solved here is scheduling, not device-ID gating. It is
not Proxmox-specific, though Proxmox is the obvious host for it and the kernel caveat above applies
there too.
I am not answering questions about this. Not in issues, not by email, not in DMs.
That is not hostility, it is capacity. Everything I know is already written down here, in more detail than I could reproduce in a reply:
- "How do I set it up?" - docs/TUTORIAL.md, thirteen parts, start to finish.
- "Will it work on my card / kernel / driver?" - docs/HARDWARE-SCOPE.md and COMPATIBILITY.md. If your combination is not listed, the honest answer is that nobody knows.
- "Where do I get the driver?" - NVIDIA's licensing portal. I cannot give you a copy and will not respond to requests for one.
- "It does not work / I get error X." - the troubleshooting table covers the failures that were actually hit here. Beyond that, read notes/WHAT-DIDNT-WORK.md before asking anyone anything, because roughly 80 approaches were already measured and most of them failed.
- "Can you add support for ...?" - no. Use the method, see docs/FINDING-THE-ISR-OFFSET.md.
- "Is this legal / allowed?" - ask a lawyer, not me.
Issues are open for exactly two things: a row for COMPATIBILITY.md (including a failure - those are just as useful) and a concrete bug with a reproduction. Pull requests are welcome and will be read. Questions will be closed unanswered.
If the documentation is wrong or unclear, that is worth a pull request fixing it, which helps more than an answer to one person would.
Free to use, all of it. Scripts, kernel modules, documentation, the measurements, the dead ends. Your project, your article, your product, commercial or not. Modify it, redistribute it, no permission needed.
Credit "Nium", with a link back here where that is reasonable. That is the only condition.
Formally: kmod/ is GPL-2.0-only because Linux kernel modules calling GPL-only exports cannot be
anything else; scripts/, lib/ and host-config/ are MIT; the documentation is CC-BY-4.0. All
three require attribution, so crediting Nium satisfies all of them. Full text and the reasoning:
LICENSE.
No warranty, and read that part seriously - this loads kernel modules, resets PCI devices and places kprobes at hard-coded offsets in a proprietary driver. It wedged a GPU and oopsed the host more than once during development. See Part 11 of the tutorial first.
This took a long time and the result is given away. If it saved you some of that time and you feel like it, Monero:
47BzeT42HzDArmPkUTrCQ2D1VidW4LiwA9L4E3k5ENrZVN7WPTx8yw2NHLcmQY7b9JBEgvvNSWUmqPmmGXdpLe6Y8QKniEs
Entirely optional. Nothing here is gated behind it.