Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions docs/developers/mps-v3-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
title: NVIDIA MPS v3 backend design
description: Architecture, GPU scheduling, allocation, recovery, and cleanup for the HAMi MPS v3 backend.
sidebar_label: MPS v3 design
---

HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples.

## Architecture

```mermaid
flowchart TB
subgraph HAMi["HAMi internals"]
direction LR
S["Scheduler"] --> D["Device plugin"]
D --> M["MPS manager"]
M <--> ST["Node state"]
D --> G["Physical GPU CDI"]
M --> A["Allocation CDI"]
end
subgraph External["External components"]
direction LR
N["MPS daemon"]
R["Container runtime"]
R --> W["CUDA application"]
W -->|namespace socket| N
end
M -->|MPS control CLI| N
A --> R
G --> R
D -.->|CDI device ID| R
style HAMi fill:#eaf2ff,stroke:#4073ad,color:#172e4a
style External fill:#e8f5ee,stroke:#3f805d,color:#183c2a
classDef internal fill:#f7faff,stroke:#4073ad,color:#172e4a
classDef external fill:#f5fcf7,stroke:#3f805d,color:#183c2a
class S,D,M,ST,A internal
class N,G,R,W external
```

The scheduler matches the node strategy and accounts for each GPU. The device plugin reads allocation annotations, and the MPS manager checks resource ownership and records allocation intent and completion on the node. HAMi generates the allocation CDI with a read-only socket mount and environment variables for routing and memory. The container runtime applies its mount and environment edits, and the CUDA client connects to the NVIDIA MPS namespace through the mounted socket. The device plugin uses NVIDIA Container Toolkit libraries to generate and publish the physical GPU CDI. The runtime invokes the memory hook specified in that CDI.

## Scheduling contract

The plugin reports `mps-v3`, the configured compute strategy, SM geometry, and allocated GPU sets. The scheduler matches `hami.io/mps-compute-mode` against the reported strategy, with `namespace-cap` as the Pod default. Both strategies use the existing per-GPU 100-unit compute budget and quota accounting. When core parsing returns zero, the backend charges 100 units for a percentage or timeslice request.

| Strategy | Server identity | Runtime provisioning |
| --- | --- | --- |
| `namespace-cap` | Sorted GPU UUID set | One namespace per Pod UID and container name. HAMi sets its active-thread percentage or assigns a static SM partition on every GPU. |
| `server-timeslice` | Pod UID, container name, and sorted GPU UUID set | One server per GPU container. HAMi programs and reads back `timeslice-pct` before the container starts and sets the namespace percentage to 100. |

Each container uses the same memory and compute allocation on every requested GPU. Identical GPU sets can share a server in `namespace-cap` and can have independent servers in `server-timeslice`. Partial overlaps are rejected by scheduling and checked again by the manager. Percentage and static SM provisioning cannot share a GPU.

Static partition accounting uses the base SM count. Before creating static partitions, the manager checks the driver-reported total SMs and chunks against the scheduler geometry. The scheduler charges `ceil(base SMs / total SMs × 100)`. If lending or borrowing is enabled, the base must consist of whole chunks. The finite maximum must cover the base and fit the GPU. The driver's runtime borrowing decisions do not change scheduler reservations.

## Allocation and runtime

### Allocation flow

```mermaid
flowchart TD
V["Validate request"] --> I["Record intent"]
I --> B{"Compute strategy"}
B -->|server-timeslice| T["Create weighted server"]
B -->|namespace-cap| C["Get shared server"]
T --> TN["Create namespace"]
C --> CN["Configure namespace"]
TN --> Q["Verify allocation"]
CN --> Q
Q --> P["Publish CDI and save state"]
P --> A["Return Allocate response"]
T -.->|Operation failed| X["Roll back"]
C -.-> X
TN -.-> X
CN -.-> X
Q -.-> X
P -.-> X
```

1. Resolve the GPU set and check the allocation's strategy, memory, and compute settings. Check that any server to be reused has recorded ownership.
2. Persist the allocation intent with stable resource names before creating any resources.
3. Create or reuse the server. Set the weight for a new timeslice server before creating the namespace. For static SM requests, create each GPU's partition and bind it to the namespace using a separate `namespace set --device=<uuid> --sm-partition=<name>` operation.
4. Check the server's GPU set, namespace percentage, timeslice weight where applicable, and the socket. Read `sm-partitions` by GPU UUID and check the underlying partition configuration. Recovery checks base chunks, remainder, lending and borrowing parameters, and binding. Changes to effective SM capacity during borrowing are expected.
5. Publish the allocation's CDI entry and persist the completed state before returning the allocation. If any container in the request fails, roll back the resources newly created for that request.

The allocation-specific CDI entry supplies the read-only namespace socket mount, `CUDA_MPS_PIPE_DIRECTORY`, and integer MiB values for `NVIDIA_GPU_MEMORY_REQUEST` and `NVIDIA_GPU_MEMORY_LIMIT`. The physical GPU's CDI entry invokes the Toolkit hook during `createRuntime`. The namespace mount exposes only the allocated namespace. Static partition selection is handled by the server, which takes precedence over a client-provided partition identifier. [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments)

Compute routing uses the namespace socket. A future cgroup routing integration requires a Toolkit interface that can bind the real container cgroup before startup, refresh the mapping on restart, and remove it on exit. Startup must fail if that mapping fails, and the memory hook must still run. [NVIDIA cgroup matching](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#cgroup-matching)

## Daemon, recovery, and cleanup

The manager takes an exclusive lock on the dedicated state directory. It records the state version, Pod and container identity, GPU set, limits, SM options, resource names, and preparation status in `allocations.json`. It uses the node's Pod list to reconcile these records with the daemon.

On a drained daemon, the manager sets `compute-policy` and verifies the result. For `server-timeslice`, the plugin checks each GPU through NVML and makes only GPUs in `DEFAULT` compute mode eligible. The manager validates the actual GPU set when creating a workload server. Restoring the default yield requires the operator to drain the node and restart the host daemon. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing)

Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Daemon or global policy failures mark all GPUs unhealthy. An allocation health check failure affects only the GPUs in that allocation. Read-only health checks run outside the allocation locks.

Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. The host service owns the daemon lifecycle. If the host daemon exits, affected workloads require recovery.

Reclamation deletes the namespace without force, deletes its partitions, deletes the server once it has no remaining allocations, then removes the allocation's CDI entry and state. r620 refuses namespace deletion while clients are attached. Failed cleanup retains its record for retry. Normal container restarts keep their allocation because kubelet can reuse its Allocate response.
162 changes: 162 additions & 0 deletions docs/userguide/nvidia-device/mps-v3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
title: Share NVIDIA GPUs with MPS v3
description: Configure MPS v3 in HAMi and request GPU memory, compute caps, static SM partitions, or weighted timeslices.
sidebar_label: MPS v3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a description field to all four new pages

---

The `mps-v3` backend uses the NVIDIA Container Toolkit CDI hook to apply each container's GPU memory limit to its cgroup. CUDA processes connect to their assigned MPS namespace for compute sharing. This backend uses the existing HAMi GPU, memory, and core resources.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hami master has no mps-v3, open that pr first


The HAMi implementation is tracked in [HAMi PR #3443](https://github.com/Project-HAMi/HAMi/pull/3443).

## NVIDIA device plugin compatibility

HAMi enables r620 MPS v3 with `operatingmode: mps-v3`. It supports per-container GPU memory limits through CDI and offers namespace caps, static SM partitions with optional borrowing, and weighted server timeslices.

The upstream NVIDIA device plugin's experimental `sharing.mps` uses Legacy MPS to expose equal-sized GPU replicas. That setting is unavailable in HAMi, where the node operating mode for MPS is `mps-v3`. [NVIDIA device plugin MPS documentation](https://github.com/NVIDIA/k8s-device-plugin/blob/v0.20.1/README.md#with-cuda-mps)

## Prepare the node

- Use Linux with cgroup v2, NVIDIA driver **r620**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver.
- Disable MIG before enrolling the node. Configure a CDI-capable container runtime and UUID device IDs. Set device memory and core scaling to `1` on MPS v3 nodes.
- Make the driver-matched `nvidia-cuda-mps-control` and its libraries available inside the privileged device plugin, which uses the host PID namespace. Set `devicePlugin.mpsV3.binary` to the path visible inside the plugin. The chart's host `/usr/bin` mount is visible at `/usrbin`. With GPU Operator, locate the binary in the configured driver installation.
- Verify that the published CDI device for each GPU includes the memory hook and that the hook path is executable on the host. HAMi checks the published spec before registering devices. The Toolkit and driver manage the memory controller. [NVIDIA memory partitioning](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html)

```bash
stat -fc %T /sys/fs/cgroup
nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv
```

## Run the MPS daemon on the host

Start a dedicated daemon on each MPS v3 node before enabling the device plugin. This example systemd unit uses the host binary path. The directories must match the Chart setting `devicePlugin.mpsV3.root`.

```ini
[Unit]
Description=HAMi NVIDIA MPS v3 daemon

[Service]
Type=simple
Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe
Environment=CUDA_MPS_LOG_DIRECTORY=/var/run/hami/mps-v3/log
ExecStartPre=/usr/bin/mkdir -p /var/run/hami/mps-v3/pipe /var/run/hami/mps-v3/log
ExecStart=/usr/bin/nvidia-cuda-mps-control -p 3 -f -m

[Install]
WantedBy=multi-user.target
```

Save the unit as `/etc/systemd/system/hami-mps-v3.service` and run the following on the node.

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now hami-mps-v3.service
sudo env CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe \
nvidia-cuda-mps-control -p 3 compute-policy get timeslice
```

The device plugin connects to the running daemon. Drain MPS workloads before stopping or restarting the host service.

## Configure a node

Select the node and choose its compute strategy in the Helm overrides. This example enables `server-timeslice` on one node.

```yaml
devicePlugin:
hostPID: true
deviceListStrategy: cdi-cri
mpsV3:
enabled: true
binary: /usrbin/nvidia-cuda-mps-control
nodeConfiguration:
config: |
{
"nodeconfig": [
{
"name": "gpu-node-01",
"operatingmode": "mps-v3",
"devicememoryscaling": 1,
"devicecorescaling": 1,
"mpsv3": {
"computeMode": "server-timeslice",
"timeslice": "long",
"timesliceYieldLevel": 1
}
}
]
}
```

To select the default `namespace-cap` strategy, omit the `mpsv3` object or use `{"computeMode":"namespace-cap"}`. Each selected `mpsv3` object replaces the previous one. Enable `mpsV3.enabled` to mount the node-local state and pipe directories. Choose `cdi-cri` when the kubelet and CRI support CDI devices, or `cdi-annotations` when the runtime consumes CDI annotations. Use only CDI strategies on these nodes.

## Request compute and memory

Set `nvidia.com/vgpu-mode: mps-v3` on the Pod. The optional `hami.io/mps-compute-mode` annotation selects the node's compute strategy and defaults to `namespace-cap`. Both strategies reserve core units from a 100-unit budget on each GPU. A resolved `gpucores` value of zero reserves 100 units. Use `nvidia.com/gpumem` or `nvidia.com/gpumem-percentage` according to the [GPU memory request rules](./specify-device-memory-usage.md). The Toolkit receives the final allocation in MiB as both the request and the hard limit.

| Strategy | Meaning of `nvidia.com/gpucores: 25` |
| --- | --- |
| `namespace-cap` | The namespace uses an active-thread percentage of 25. CUDA may round the available compute resources to a hardware-supported amount. |
| `server-timeslice` | The container's server uses 25% of the daemon's base compute timeslice. Actual GPU time and throughput depend on which servers have work ready to run. |

For example, this Pod uses 4 GiB of GPU memory and a timeslice of 25% of the configured base duration.

```yaml
apiVersion: v1
kind: Pod
metadata:
name: mps-training
annotations:
nvidia.com/vgpu-mode: mps-v3
hami.io/mps-compute-mode: server-timeslice
spec:
restartPolicy: Never
containers:
- name: training
image: your-cuda-application:tag
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 4096
nvidia.com/gpucores: 25
```

Use `nvidia.com/gpu: 2` for two GPUs. Each GPU in a container must receive the same final memory and compute allocation. GPU sets can be identical or disjoint across containers. A request for GPUs A and B conflicts with an existing allocation for A alone.

For `server-timeslice`, `mpsv3.timeslice` accepts `default` (the device's default), `short` (10 ms), `medium` (20 ms), or `long` (50 ms). The optional `mpsv3.timesliceYieldLevel` accepts `1` or `2`. The driver may reject percentages that produce a slice below the device's minimum. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing)

## Request static SMs and optional borrowing

For a `namespace-cap` Pod, add the following annotations and omit `gpucores` if the SM count is the only compute request.

```yaml
metadata:
annotations:
nvidia.com/vgpu-mode: mps-v3
hami.io/mps-sm-count: "16"
hami.io/mps-sm-shareable: "true"
hami.io/mps-sm-max-chunks: "16"
```

- `hami.io/mps-sm-count` applies to each GPU of every GPU container in the Pod. An Ampere discrete GPU uses 2 SMs per chunk, so the example reserves eight chunks. Hopper and newer discrete GPUs use 8 SMs per chunk. Integrated GPUs use 2 SMs per chunk. A fixed partition can also use the device's remainder if it yields the exact requested count.
- `hami.io/mps-sm-shareable` allows the driver to lend the partition's idle SMs while it has no attached clients. Its default is `false`.
- `hami.io/mps-sm-max-chunks` caps the **total** chunks the partition may use, including its base reservation. The default `0` disables borrowing. A positive integer must be between the base and total chunks on every allocated GPU. Use `unlimited` to let the driver use available capacity without a configured cap.
- Borrowing or lending requires a base reservation made of whole chunks. The scheduler and quota account for the base reservation as `ceil(requested SMs / total SMs × 100)`. An explicit `gpucores` value must cover this charge. The driver controls how many additional SMs can be used at runtime.

HAMi binds each partition to the namespace by GPU UUID. The namespace uses an active-thread percentage of 100 in this case. A physical GPU can serve either static SM partitions or namespace percentage allocations at a given time. The driver's ability to create another partition with clients already attached depends on the environment. HAMi returns the driver's error and rolls back the attempted allocation if creation fails. [NVIDIA partition load balancing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#sm-partition-load-balancing), [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments)

## Upgrade, switch strategy, or diagnose a failure

1. Prevent new workloads from landing on the node and drain its MPS v3 workloads. For an upgrade from the earlier backend, let the old manager reclaim its resources before replacing it.
2. Apply the new driver, plugin, or node policy after reclamation. The same drain applies before changing `computeMode`, the base timeslice, the yield level, or the state root.
3. Confirm that the plugin reports healthy devices and run a workload with the intended strategy before reopening the node to GPU workloads.

| Symptom | What to check |
| --- | --- |
| The node rejects MPS v3 during startup | Check driver r620, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. |
| A Pod cannot find a compatible GPU | Check the Pod and node strategies, available core and memory budget, the SM chunk geometry, and existing GPU sets. |
| Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. Each GPU must use `DEFAULT` compute mode for `server-timeslice`. |
| Cleanup is waiting for clients | Let clients exit. HAMi removes the namespace first, then the partition and the last unused server. |
| The daemon is lost with live allocations | Investigate the daemon and restart affected workloads through Kubernetes. Preserve the allocation state while the Pods are live. |

Each container receives a read-only mount of its namespace socket directory at `/run/hami-mps`. Compute settings apply to CUDA processes that attach through that socket. The MPS policies do not isolate memory bandwidth or provide MIG-style fault containment. Container restarts retain their allocations. Manager recovery requires the daemon and its resources to be available for live workloads.

See the [MPS v3 backend design](../../developers/mps-v3-design.md) for allocation and recovery details.
Loading
Loading