Skip to content

About

Device-plugin for volcano vgpu which support hard resource isolation

Resources

Code of conduct

Contributing

Security policy

Stars

178 stars

Watchers

3 watching

Forks

Volcano vGPU device plugin

A Kubernetes device plugin that lets pods share NVIDIA GPUs under the Volcano scheduler. It is a subproject of HAMi, based on the NVIDIA device plugin, and uses HAMi-core to enforce per-container memory and core limits.

A GPU is split into several vGPUs. Each pod asks for a number of vGPUs and, optionally, device memory and a share of the cores. There are two sharing modes:

  • hami-core: software isolation, works on any GPU.
  • mig: dynamic MIG, hardware isolation, needs an Ampere or newer GPU.

A node uses one mode, and a cluster can mix both. Plugin settings are described in doc/config.md.

Compatibility

volcano-vgpu Volcano
v1.13.0 v1.15.0 and above
v1.12.0 and below v1.15.0 and below
master master

The plugin is built with client-go 0.37. The manifests in this repository were validated against Kubernetes 1.36.

Requirements

  • An NVIDIA driver and the NVIDIA Container Toolkit on every GPU node, with nvidia as the default container runtime.
  • The Volcano scheduler with vGPU enabled (below).

Enable vGPU in Volcano

In volcano-scheduler-configmap (namespace volcano-system), add the deviceshare plugin with VGPUEnable:

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
  - name: conformance
- plugins:
  - name: drf
  - name: deviceshare
    arguments:
      deviceshare.VGPUEnable: true
      deviceshare.SchedulePolicy: binpack  # or spread
  - name: predicates
  - name: proportion
  - name: nodeorder
  - name: binpack

Install

No Helm repository is published yet, so install from a checkout:

git clone https://github.com/Project-HAMi/volcano-vgpu-device-plugin
cd volcano-vgpu-device-plugin
helm install volcano-vgpu-device-plugin deployments/helm/volcano-vgpu-device-plugin

Or apply the plain manifest:

kubectl apply -f deployments/static/volcano-vgpu-device-plugin.yml

Useful chart options:

  • --set cdi.enabled=true passes devices through CDI instead of environment variables.
  • --set nvmlMock.enabled=true runs against nvml-mock on a cluster without GPUs, which is enough to check scheduling and registration. It cannot be combined with cdi.enabled.

Check that the node reports vGPUs:

kubectl get node <node> -o jsonpath='{.status.allocatable}'

volcano.sh/vgpu-number and volcano.sh/vgpu-memory should be listed. The number is the GPU count times deviceSplitCount (10 by default).

Run a vGPU job

Request vGPUs in resources.limits:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  annotations:
    volcano.sh/vgpu-mode: "hami-core"  # optional: hami-core or mig
spec:
  schedulerName: volcano
  containers:
  - name: cuda
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["sleep", "infinity"]
    resources:
      limits:
        volcano.sh/vgpu-number: 2    # two vGPUs
        volcano.sh/vgpu-memory: 3000 # optional: MB per vGPU
        volcano.sh/vgpu-cores: 50    # optional: percent of a GPU per vGPU

In hami-core mode nvidia-smi inside the container shows the vgpu-memory limit. In mig mode it shows the memory of the assigned MIG instance, which comes from the configured MIG geometry. A pod cannot request more vGPUs than the node has GPUs. Without the annotation, either mode may be used.

Metrics

The plugin pod serves per-container GPU usage and limits on port 9394:

curl <plugin-pod-ip>:9394/metrics

Contributing and security

See CONTRIBUTING.md. Report vulnerabilities as described in SECURITY.md, not in a public issue. Questions go to GitHub issues or the HAMi community.

License

Apache-2.0, see LICENSE.

About

Device-plugin for volcano vgpu which support hard resource isolation

Resources

Code of conduct

Contributing

Security policy

Stars

178 stars

Watchers

3 watching

Forks

Releases

Packages

Used by

Contributors

Languages