A Kubernetes device plugin that lets pods share NVIDIA GPUs under the Volcano scheduler. It is a subproject of HAMi, based on the NVIDIA device plugin, and uses HAMi-core to enforce per-container memory and core limits.
A GPU is split into several vGPUs. Each pod asks for a number of vGPUs and, optionally, device memory and a share of the cores. There are two sharing modes:
hami-core: software isolation, works on any GPU.mig: dynamic MIG, hardware isolation, needs an Ampere or newer GPU.
A node uses one mode, and a cluster can mix both. Plugin settings are described in doc/config.md.
| volcano-vgpu | Volcano |
|---|---|
| v1.13.0 | v1.15.0 and above |
| v1.12.0 and below | v1.15.0 and below |
| master | master |
The plugin is built with client-go 0.37. The manifests in this repository were validated against Kubernetes 1.36.
- An NVIDIA driver and the NVIDIA Container Toolkit on every GPU node, with
nvidiaas the default container runtime. - The Volcano scheduler with vGPU enabled (below).
In volcano-scheduler-configmap (namespace volcano-system), add the deviceshare plugin with VGPUEnable:
actions: "enqueue, allocate, backfill"
tiers:
- plugins:
- name: priority
- name: gang
- name: conformance
- plugins:
- name: drf
- name: deviceshare
arguments:
deviceshare.VGPUEnable: true
deviceshare.SchedulePolicy: binpack # or spread
- name: predicates
- name: proportion
- name: nodeorder
- name: binpackNo Helm repository is published yet, so install from a checkout:
git clone https://github.com/Project-HAMi/volcano-vgpu-device-plugin
cd volcano-vgpu-device-plugin
helm install volcano-vgpu-device-plugin deployments/helm/volcano-vgpu-device-pluginOr apply the plain manifest:
kubectl apply -f deployments/static/volcano-vgpu-device-plugin.ymlUseful chart options:
--set cdi.enabled=truepasses devices through CDI instead of environment variables.--set nvmlMock.enabled=trueruns against nvml-mock on a cluster without GPUs, which is enough to check scheduling and registration. It cannot be combined withcdi.enabled.
Check that the node reports vGPUs:
kubectl get node <node> -o jsonpath='{.status.allocatable}'volcano.sh/vgpu-number and volcano.sh/vgpu-memory should be listed. The number is the GPU count times deviceSplitCount (10 by default).
Request vGPUs in resources.limits:
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
annotations:
volcano.sh/vgpu-mode: "hami-core" # optional: hami-core or mig
spec:
schedulerName: volcano
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base-ubuntu22.04
command: ["sleep", "infinity"]
resources:
limits:
volcano.sh/vgpu-number: 2 # two vGPUs
volcano.sh/vgpu-memory: 3000 # optional: MB per vGPU
volcano.sh/vgpu-cores: 50 # optional: percent of a GPU per vGPUIn hami-core mode nvidia-smi inside the container shows the vgpu-memory limit. In mig mode it shows the memory of the assigned MIG instance, which comes from the configured MIG geometry. A pod cannot request more vGPUs than the node has GPUs. Without the annotation, either mode may be used.
The plugin pod serves per-container GPU usage and limits on port 9394:
curl <plugin-pod-ip>:9394/metricsSee CONTRIBUTING.md. Report vulnerabilities as described in SECURITY.md, not in a public issue. Questions go to GitHub issues or the HAMi community.
Apache-2.0, see LICENSE.