From 0dcabafa0082000ad9dfa02f97dd91d93657a1f1 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Thu, 12 Nov 2026 14:23:36 +0800 Subject: [PATCH 1/7] docs(nvidia): document the r620 MPS v3 backend Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 59 +++++++++ docs/userguide/nvidia-device/mps-v3.md | 123 ++++++++++++++++++ .../current/developers/mps-v3-design.md | 59 +++++++++ .../current/userguide/nvidia-device/mps-v3.md | 123 ++++++++++++++++++ sidebars.js | 2 + 5 files changed, 366 insertions(+) create mode 100644 docs/developers/mps-v3-design.md create mode 100644 docs/userguide/nvidia-device/mps-v3.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md new file mode 100644 index 000000000..2e40c359f --- /dev/null +++ b/docs/developers/mps-v3-design.md @@ -0,0 +1,59 @@ +--- +title: NVIDIA MPS v3 backend design +sidebar_label: MPS v3 design +--- + +HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 or later and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples. + +## Scheduling contract + +The plugin reports `mps-v3`, the configured compute strategy, SM geometry, and allocated GPU sets. The scheduler matches `hami.io/mps-compute-mode` against the reported strategy, with `namespace-cap` as the Pod default. Both strategies use the existing per-GPU 100-unit compute budget and quota accounting. When core parsing returns zero, the backend charges 100 units for a percentage or timeslice request. + +| Strategy | Server identity | Runtime provisioning | +| --- | --- | --- | +| `namespace-cap` | Sorted GPU UUID set | One namespace per Pod UID and container name. HAMi sets its active-thread percentage or assigns a static SM partition on every GPU. | +| `server-timeslice` | Pod UID, container name, and sorted GPU UUID set | One server per GPU container. HAMi programs and reads back `timeslice-pct` before the container starts and sets the namespace percentage to 100. | + +Each container uses the same memory and compute allocation on every requested GPU. Identical GPU sets can share a server in `namespace-cap` and can have independent servers in `server-timeslice`. Partial overlaps are rejected by scheduling and checked again by the manager. Percentage and static SM provisioning cannot share a GPU. + +Static partition accounting uses the base SM count. The scheduler validates each GPU's chunk geometry and charges `ceil(base SMs / total SMs × 100)`. If lending or borrowing is enabled, the base must consist of whole chunks. The finite maximum must cover the base and fit the GPU. The driver's runtime borrowing decisions do not change scheduler reservations. + +## Allocation and runtime + +1. Resolve the GPU set and check the allocation's strategy, memory, and compute settings. Check that any server to be reused has recorded ownership. +2. Persist the allocation intent with stable resource names before creating any resources. +3. Create or reuse the server. Set the weight for a new timeslice server before creating the namespace. For static SM requests, create each GPU's partition and bind it to the namespace using a separate `namespace set --device= --sm-partition=` operation. +4. Check the server's GPU set, namespace percentage, timeslice weight where applicable, and the socket. Read `sm-partitions` by GPU UUID and check the underlying partition configuration. Recovery checks base chunks, remainder, lending and borrowing parameters, and binding. Changes to effective SM capacity during borrowing are expected. +5. Publish the allocation's CDI entry and persist the completed state before returning the allocation. If any container in the request fails, roll back the resources newly created for that request. + +The allocation-specific CDI entry supplies the read-only namespace socket mount, `CUDA_MPS_PIPE_DIRECTORY`, and integer MiB values for `NVIDIA_GPU_MEMORY_REQUEST` and `NVIDIA_GPU_MEMORY_LIMIT`. The physical GPU's CDI entry invokes the Toolkit hook during `createRuntime`. The namespace mount exposes only the allocated namespace. Static partition selection is handled by the server, which takes precedence over a client-provided partition identifier. [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +Compute routing uses the namespace socket. A future cgroup routing integration requires a Toolkit interface that can bind the real container cgroup before startup, refresh the mapping on restart, and remove it on exit. Startup must fail if that mapping fails, and the memory hook must still run. [NVIDIA cgroup matching](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#cgroup-matching) + +## Daemon, recovery, and cleanup + +The manager takes an exclusive lock on the dedicated state directory. It records the state version, Pod and container identity, GPU set, limits, SM options, resource names, and preparation status in `allocations.json`. It uses the node's Pod list to reconcile these records with the daemon. + +On a drained daemon, the manager sets `compute-policy` and verifies the result. A drained `server-timeslice` node probes two concurrently ready servers for the configured GPU set. If yield was previously set and the desired value is omitted, the manager restarts the drained daemon to restore the driver default. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Failed daemon or allocation health checks mark devices unhealthy and block new allocations. + +Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. If a plugin container restart also stops the daemon, affected workloads require recovery. + +Reclamation deletes the namespace without force, deletes its partitions, deletes the server once it has no remaining allocations, then removes the allocation's CDI entry and state. r620 refuses namespace deletion while clients are attached. Failed cleanup retains its record for retry. Normal container restarts keep their allocation because kubelet can reuse its Allocate response. + +## Verification on A10 + +The following results were observed on driver 620.70.06. + +| Check | Observed result | +| --- | --- | +| Two servers configured at 25% and 75% with a long base timeslice | Both settings read back and both servers ran on the same GPU. | +| Continuously queued CUDA Graph workload | Median GPU context residence was 12.9 ms and 38.4 ms. Throughput was approximately 1,024 and 2,816–3,072 kernels/s, respectively. | +| Synchronous short kernels | The two workloads had nearly equal GPU context residence and throughput. The remaining server returned to its single-workload throughput after competitors stopped. | +| SM borrowing with 32 base SMs | The borrower executed across 64 SMs with the lender idle, 40 after the lender connected, and 64 after it exited. The reported base count stayed at 32. | +| Static SM binding | A namespace assigned 16 SMs used that partition even when the client requested a different 4-SM partition through its environment. | +| Toolkit 1.20.1 CDI hook | Two separate container starts saw a 1 GiB limit and returned OOM for an oversized allocation, including after changing the in-process environment. | +| Lifecycle | Validated manager reattachment, policy changes after draining, state preservation after daemon loss, and refusal to delete a namespace with active clients. | + +These measurements characterize the tested GPU and workload. Physical multi-GPU and Kubernetes end-to-end coverage remain separate rollout checks. The Kubernetes installation on the test host used Toolkit 1.17.8. The CDI memory test used a separate Toolkit 1.20.1 installation. diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md new file mode 100644 index 000000000..3c84f74b9 --- /dev/null +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -0,0 +1,123 @@ +--- +title: Share NVIDIA GPUs with MPS v3 +sidebar_label: MPS v3 +--- + +The `mps-v3` backend uses the NVIDIA Container Toolkit CDI hook to apply each container's GPU memory limit to its cgroup. CUDA processes connect to their assigned MPS namespace for compute sharing. This backend uses the existing HAMi GPU, memory, and core resources. + +## Prepare the node + +- Use Linux with cgroup v2, NVIDIA driver **r620 or later**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver. +- Disable MIG before enrolling the node. Configure a CDI-capable container runtime and UUID device IDs. Set device memory and core scaling to `1` on MPS v3 nodes. +- Make the driver-matched `nvidia-cuda-mps-control` and its libraries available inside the privileged device plugin, which uses the host PID namespace. Set `devicePlugin.mpsV3.binary` to the path visible inside the plugin. The chart's host `/usr/bin` mount is visible at `/usrbin`. With GPU Operator, locate the binary in the configured driver installation. +- Verify that the published CDI device for each GPU includes the memory hook and that the hook path is executable on the host. HAMi checks the published spec before registering devices. The Toolkit and driver manage the memory controller. [NVIDIA memory partitioning](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) + +```bash +stat -fc %T /sys/fs/cgroup +nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv +``` + +## Configure a node + +Select the node and choose its compute strategy in the Helm overrides. This example enables `server-timeslice` on one node. + +```yaml +devicePlugin: + hostPID: true + deviceListStrategy: cdi-cri + mpsV3: + enabled: true + binary: /usrbin/nvidia-cuda-mps-control + nodeConfiguration: + config: | + { + "nodeconfig": [ + { + "name": "gpu-node-01", + "operatingmode": "mps-v3", + "devicememoryscaling": 1, + "devicecorescaling": 1, + "mpsv3": { + "computeMode": "server-timeslice", + "timeslice": "long", + "timesliceYieldLevel": 1 + } + } + ] + } +``` + +To select the default `namespace-cap` strategy, omit the `mpsv3` object or use `{"computeMode":"namespace-cap"}`. Each selected `mpsv3` object replaces the previous one. Enable `mpsV3.enabled` to mount the node-local state and pipe directories. Choose `cdi-cri` when the kubelet and CRI support CDI devices, or `cdi-annotations` when the runtime consumes CDI annotations. Use only CDI strategies on these nodes. + +## Request compute and memory + +Set `nvidia.com/vgpu-mode: mps-v3` on the Pod. The optional `hami.io/mps-compute-mode` annotation selects the node's compute strategy and defaults to `namespace-cap`. Both strategies reserve core units from a 100-unit budget on each GPU. A resolved `gpucores` value of zero reserves 100 units. Use `nvidia.com/gpumem` or `nvidia.com/gpumem-percentage` according to the [GPU memory request rules](./specify-device-memory-usage.md). The Toolkit receives the final allocation in MiB as both the request and the hard limit. + +| Strategy | Meaning of `nvidia.com/gpucores: 25` | +| --- | --- | +| `namespace-cap` | The namespace uses an active-thread percentage of 25. CUDA may round the available compute resources to a hardware-supported amount. | +| `server-timeslice` | The container's server uses 25% of the daemon's base compute timeslice. Actual GPU time and throughput depend on which servers have work ready to run. | + +For example, this Pod uses 4 GiB of GPU memory and a timeslice of 25% of the configured base duration. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mps-training + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-compute-mode: server-timeslice +spec: + restartPolicy: Never + containers: + - name: training + image: your-cuda-application:tag + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 25 +``` + +Use `nvidia.com/gpu: 2` for two GPUs. Each GPU in a container must receive the same final memory and compute allocation. GPU sets can be identical or disjoint across containers. A request for GPUs A and B conflicts with an existing allocation for A alone. + +For `server-timeslice`, `mpsv3.timeslice` accepts `default` (the device's default), `short` (10 ms), `medium` (20 ms), or `long` (50 ms). The optional `mpsv3.timesliceYieldLevel` accepts `1` or `2`. The driver may reject percentages that produce a slice below the device's minimum. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +## Request static SMs and optional borrowing + +For a `namespace-cap` Pod, add the following annotations and omit `gpucores` if the SM count is the only compute request. + +```yaml +metadata: + annotations: + nvidia.com/vgpu-mode: mps-v3 + nvidia.com/mps-sm-count: "16" + hami.io/mps-sm-shareable: "true" + hami.io/mps-sm-max-chunks: "16" +``` + +- `nvidia.com/mps-sm-count` applies to each GPU of every GPU container in the Pod. An Ampere discrete GPU uses 2 SMs per chunk, so the example reserves eight chunks. Hopper and newer discrete GPUs use 8 SMs per chunk. Integrated GPUs use 2 SMs per chunk. A fixed partition can also use the device's remainder if it yields the exact requested count. +- `hami.io/mps-sm-shareable` allows the driver to lend the partition's idle SMs while it has no attached clients. Its default is `false`. +- `hami.io/mps-sm-max-chunks` caps the **total** chunks the partition may use, including its base reservation. The default `0` disables borrowing. A positive integer must be between the base and total chunks on every allocated GPU. Use `unlimited` to let the driver use available capacity without a configured cap. +- Borrowing or lending requires a base reservation made of whole chunks. The scheduler and quota account for the base reservation as `ceil(requested SMs / total SMs × 100)`. An explicit `gpucores` value must cover this charge. The driver controls how many additional SMs can be used at runtime. + +HAMi binds each partition to the namespace by GPU UUID. The namespace uses an active-thread percentage of 100 in this case. A physical GPU can serve either static SM partitions or namespace percentage allocations at a given time. The driver's ability to create another partition with clients already attached depends on the environment. HAMi returns the driver's error and rolls back the attempted allocation if creation fails. [NVIDIA partition load balancing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#sm-partition-load-balancing), [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +## Upgrade, switch strategy, or diagnose a failure + +1. Prevent new workloads from landing on the node and drain its MPS v3 workloads. For an upgrade from the earlier backend, let the old manager reclaim its resources before replacing it. +2. Apply the new driver, plugin, or node policy after reclamation. The same drain applies before changing `computeMode`, the base timeslice, the yield level, or the state root. +3. Confirm that the plugin reports healthy devices and run a workload with the intended strategy before reopening the node to GPU workloads. + +| Symptom | What to check | +| --- | --- | +| The node rejects MPS v3 during startup | Check driver r620 or later, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. | +| A Pod cannot find a compatible GPU | Check the Pod and node strategies, available core and memory budget, the SM chunk geometry, and existing GPU sets. | +| Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. The plugin probes two concurrent servers on a drained node before advertising the strategy. | +| Cleanup is waiting for clients | Let clients exit. HAMi removes the namespace first, then the partition and the last unused server. | +| The daemon is lost with live allocations | Investigate the daemon and restart affected workloads through Kubernetes. Preserve the allocation state while the Pods are live. | + +Each container receives a read-only mount of its namespace socket directory at `/run/hami-mps`. Compute settings apply to CUDA processes that attach through that socket. The MPS policies do not isolate memory bandwidth or provide MIG-style fault containment. Container restarts retain their allocations. Manager recovery requires the daemon and its resources to be available for live workloads. + +See the [MPS v3 backend design](../../developers/mps-v3-design.md) for allocation, recovery, and hardware verification details. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md new file mode 100644 index 000000000..39272025f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -0,0 +1,59 @@ +--- +title: NVIDIA MPS v3 后端设计 +sidebar_label: MPS v3 设计 +--- + +HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限制 GPU 显存,并通过 MPS v3 共享算力。后端以 r620 为最低驱动版本,启动时检查专用 daemon 的命令接口。部署和工作负载示例见[用户指南](../userguide/nvidia-device/mps-v3.md)。 + +## 调度契约 + +设备插件上报 `mps-v3`、节点算力策略、SM 几何信息和已分配的 GPU 集合。调度器按 `hami.io/mps-compute-mode` 匹配节点策略,Pod 缺省为 `namespace-cap`。两种策略沿用每卡 100 单位的算力预算和现有配额。百分比或时间片请求的算力解析结果为 0 时,按 100 单位记账。 + +| 策略 | Server 身份 | 运行时配置 | +| --- | --- | --- | +| `namespace-cap` | 排序后的 GPU UUID 集合 | 每个 Pod UID 和容器名对应一个 namespace,设置活跃线程百分比或逐卡绑定静态分区。 | +| `server-timeslice` | Pod UID、容器名和排序后的 GPU UUID 集合 | 每个 GPU 容器使用独立 server。容器启动前写入并回读 `timeslice-pct`,namespace 百分比设为 100。 | + +同一容器在每张卡上使用相同的显存和算力额度。相同 GPU 集合在 `namespace-cap` 下可共用 server,在 `server-timeslice` 下可运行多个独立 server。调度器和管理器均拒绝部分重叠的集合。同一 GPU 不能混用静态 SM 分区和 namespace 百分比。 + +静态分区以基础 SM 数核算。调度器校验每卡的 chunk 几何,并按 `ceil(基础 SM 数 / 总 SM 数 × 100)` 记账。开启借出或借入时,基础额度须由整 chunk 组成,有限峰值须覆盖基础额度并符合每卡容量。驱动在运行时决定借用量,调度预留保持基础额度。 + +## 分配与容器启动 + +1. 校验 GPU 集合、策略、显存和算力设置。复用 server 前验证其归属。 +2. 在创建资源前将稳定的资源名称和分配意图写入节点状态。 +3. 创建或复用 server。新时间片 server 先设置权重,再创建 namespace。静态 SM 请求先按卡创建分区,再逐卡单独执行 `namespace set --device= --sm-partition=`。 +4. 回读 GPU 集合、namespace 百分比、时间片权重和 SM 绑定,检查 socket。静态分区校验覆盖基础 chunks、余数、借出和借入选项。借用造成的有效 SM 数变化属于正常运行行为。 +5. 发布专属 CDI 并持久化准备完成状态,再返回分配结果。请求内任一容器分配失败时,回滚本次新建的资源。 + +分配专属 CDI 提供只读 namespace socket 挂载、`CUDA_MPS_PIPE_DIRECTORY` 和整数 MiB 的 `NVIDIA_GPU_MEMORY_REQUEST`、`NVIDIA_GPU_MEMORY_LIMIT`。物理 GPU CDI 的 `createRuntime` hook 在应用启动前配置显存上限。服务端的静态分区绑定优先于客户端设置的分区标识。[NVIDIA namespace 绑定](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +本版通过专属 namespace socket 路由算力。接入 cgroup 路由需要 Toolkit 提供正式接口,在容器启动前绑定真实 cgroup、重启时更新映射、退出时清理。绑定失败须阻止启动,并保留显存 hook 的正常执行。[NVIDIA cgroup 路由](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#cgroup-matching) + +## Daemon、恢复与回收 + +管理器对专用状态目录加独占锁,在 `allocations.json` 记录版本、Pod 与容器身份、GPU 集合、额度、SM 选项、资源名称和准备状态。启动时根据权威节点 Pod 列表和 daemon 状态恢复分配。 + +排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点用配置的 GPU 集合验证两个 server 能同时就绪。如果要恢复为默认 yield,管理器会在无 server 时重启 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +健康检查和恢复优先使用 `get` 字段回读配置。资源是否存在、静态分区的具体设置通过 r620 CSV inventory 核对。620.70.06 的 `sm-partition list --all` 在状态列前还有一列 contract timeout。未知格式、查询超时或断连会停止当前操作并保留记录。daemon 或分配健康检查失败时,插件会上报设备不健康并阻止新分配。 + +恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。整个插件容器重启若同时结束了 daemon,则需要恢复受影响的工作负载。 + +回收依次执行非强制删除 namespace、删除分区、删除无引用 server,再移除 CDI 和状态。r620 会拒绝删除仍有客户端的 namespace,失败记录保留供重试。普通容器重启保留原分配,以便 kubelet 复用 Allocate 响应。 + +## A10 真机验证 + +以下结果来自驱动 620.70.06。 + +| 项目 | 观察结果 | +| --- | --- | +| 两个 server 配置 25% / 75% 长时间片 | 权重回读成功,两个 server 在同一张卡上运行。 | +| 持续排队的 CUDA Graph | GPU 上下文驻留中位数约为 12.9 / 38.4 ms,吞吐约为 1,024 / 2,816–3,072 kernel/s。 | +| 同步短 kernel | 两个工作负载的 GPU 上下文驻留与吞吐均接近。其他 server 停止执行后,剩余负载恢复到独自运行时的吞吐。 | +| 基础额度 32 SM 的借用 | 借出方空闲时,借入方实际覆盖 64 个 SM。借出方接入后降为 40,退出后恢复到 64。CUDA 报告的基础值仍为 32。 | +| 服务端静态绑定 | namespace 绑定 16 SM 后,即使客户端通过环境变量请求另一块 4 SM 分区,仍获得 16 SM。 | +| Toolkit 1.20.1 CDI hook | 两次容器启动均看到 1 GiB 上限。进程内修改环境变量后,超额申请仍返回 OOM。 | +| 生命周期 | 验证管理器重连、排空后切换、daemon 失联后保留状态,以及驱动阻止删除有活跃客户端的 namespace。 | + +实测值取决于 GPU 和负载。物理多卡和 Kubernetes 端到端仍待验证。测试机的集群使用 Toolkit 1.17.8,CDI 显存实验使用独立的 Toolkit 1.20.1。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md new file mode 100644 index 000000000..fe4f60b9e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -0,0 +1,123 @@ +--- +title: 使用 MPS v3 共享 NVIDIA GPU +sidebar_label: MPS v3 +--- + +`mps-v3` 后端通过 NVIDIA Container Toolkit 的 CDI hook 为每个容器的 cgroup 设置显存上限。CUDA 进程连接分配给容器的 MPS namespace,按节点配置的策略共享算力。GPU 数量、显存和算力继续使用现有的 HAMi 资源名。 + +## 准备节点 + +- 使用 cgroup v2、**r620 或更新的 NVIDIA 驱动**和 **NVIDIA Container Toolkit 1.20.1 或更新版本**。启用 `apply-cuda-memory-limits` hook。应用镜像应使用与节点驱动兼容的 CUDA 运行时。 +- 启用后端前先关闭 MIG。使用支持 CDI 的容器运行时,以 UUID 标识 GPU,并将节点的显存和算力缩放系数设为 `1`。 +- 确认特权 device plugin 使用主机 PID 命名空间,并能访问与驱动配套的 `nvidia-cuda-mps-control` 和驱动库。Chart 将主机的 `/usr/bin` 挂到插件容器内的 `/usrbin`。使用 GPU Operator 时,请按驱动的实际安装位置配置二进制路径。 +- 检查已经发布的物理 GPU CDI 设备包含显存 hook,且 hook 路径在主机上可执行。HAMi 在注册设备前会检查该配置。显存控制器由 Toolkit 和驱动管理。[NVIDIA 显存分区文档](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) + +```bash +stat -fc %T /sys/fs/cgroup +nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv +``` + +## 配置节点 + +以下 Helm 配置在 `gpu-node-01` 上启用 `server-timeslice`。按实际节点名称和驱动路径修改。 + +```yaml +devicePlugin: + hostPID: true + deviceListStrategy: cdi-cri + mpsV3: + enabled: true + binary: /usrbin/nvidia-cuda-mps-control + nodeConfiguration: + config: | + { + "nodeconfig": [ + { + "name": "gpu-node-01", + "operatingmode": "mps-v3", + "devicememoryscaling": 1, + "devicecorescaling": 1, + "mpsv3": { + "computeMode": "server-timeslice", + "timeslice": "long", + "timesliceYieldLevel": 1 + } + } + ] + } +``` + +使用默认的 `namespace-cap` 时,省略 `mpsv3` 对象或写为 `{"computeMode":"namespace-cap"}`。多条匹配的节点配置按顺序应用,后面的 `mpsv3` 对象整体替换前一个。启用 `mpsV3.enabled` 后,Chart 会挂载节点状态和 socket 目录。按 kubelet 和运行时的 CDI 接入方式选择 `cdi-cri` 或 `cdi-annotations`。MPS v3 节点只接受 CDI 设备列表策略。 + +## 申请算力和显存 + +在 Pod 上设置 `nvidia.com/vgpu-mode: mps-v3`。用 `hami.io/mps-compute-mode` 选择节点的算力策略,缺省值为 `namespace-cap`。两种策略均按每卡 100 单位核算配额,解析得到的 `gpucores` 为 0 时按 100 记账。按[显存申请规则](./specify-device-memory-usage.md)使用 `nvidia.com/gpumem` 或 `nvidia.com/gpumem-percentage`。Toolkit 将最终分配的 MiB 数同时用于显存请求和硬上限。 + +| 策略 | `nvidia.com/gpucores: 25` 的含义 | +| --- | --- | +| `namespace-cap` | namespace 的活跃线程百分比上限为 25,硬件可以对可用算力取整。 | +| `server-timeslice` | 容器专属 server 的时间片长度为 daemon 基础时间片的 25%。实际 GPU 时间和吞吐取决于各 server 有多少任务就绪。 | + +以下 Pod 申请 4 GiB 显存和基础时间片长度的 25%。 + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mps-training + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-compute-mode: server-timeslice +spec: + restartPolicy: Never + containers: + - name: training + image: your-cuda-application:tag + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 25 +``` + +申请两张卡时使用 `nvidia.com/gpu: 2`。同一容器在每张卡上的最终显存和算力额度必须相同。不同容器可以使用完全相同或互不相交的 GPU 集合。若 GPU A 已被单卡分配使用,A+B 的请求会与现有分配冲突。 + +`server-timeslice` 的 `mpsv3.timeslice` 支持 `default`(设备默认值)、`short`(10 ms)、`medium`(20 ms)和 `long`(50 ms)。可选的 `mpsv3.timesliceYieldLevel` 接受 `1` 或 `2`。当计算后的时间片低于设备支持的最小值时,驱动会拒绝设置。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +## 申请静态 SM 和可选借用 + +在 `namespace-cap` Pod 上添加以下注解。仅通过 SM 数申请算力时,可省略 `gpucores`。 + +```yaml +metadata: + annotations: + nvidia.com/vgpu-mode: mps-v3 + nvidia.com/mps-sm-count: "16" + hami.io/mps-sm-shareable: "true" + hami.io/mps-sm-max-chunks: "16" +``` + +- `nvidia.com/mps-sm-count` 适用于 Pod 内每个 GPU 容器申请的每张卡。Ampere 独立 GPU 每个 chunk 为 2 SM,因此示例预留 8 个 chunk。Hopper 及更新的独立 GPU 每个 chunk 为 8 SM,集成 GPU 为 2 SM。固定分区也可以使用该设备的余数 SM,前提是能精确组成请求值。 +- `hami.io/mps-sm-shareable` 允许驱动在分区没有客户端时借出空闲 SM。默认 `false`。 +- `hami.io/mps-sm-max-chunks` 是分区能使用的**总** chunk 数上限,包含基础额度。默认 `0` 表示禁止借入。正整数上限必须在每卡的基础 chunk 数与该卡总 chunk 数之间。`unlimited` 允许驱动使用可借容量。 +- 启用借出或借入时,基础额度必须是整 chunk。调度和配额按 `ceil(基础 SM 数 / GPU 总 SM 数 × 100)` 记账。显式设置的 `gpucores` 必须覆盖该值。运行时实际借用量由驱动决定。 + +HAMi 按 GPU UUID 将分区绑定到 namespace,并将 namespace 的活跃线程百分比设为 100。同一 GPU 同时只支持静态 SM 分区或 namespace 百分比。GPU 有活跃客户端时能否创建新分区取决于驱动与设备。若驱动拒绝,HAMi 会报告错误并回滚该次分配。[NVIDIA SM 借用](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#sm-partition-load-balancing)、[NVIDIA namespace 绑定](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +## 升级、切换策略和排查 + +1. 阻止新任务进入节点,并排空其 MPS v3 工作负载。从旧版升级时,先由旧管理器回收已有分配。 +2. 确认资源已回收,再变更驱动、插件或节点策略。修改 `computeMode`、基础时间片、yield 或状态目录也需要排空。 +3. 确认插件上报健康,运行符合新策略的验证任务,再恢复节点调度。 + +| 现象 | 检查项 | +| --- | --- | +| MPS v3 节点启动失败 | 检查 r620 驱动、cgroup v2、非 MIG GPU、UUID 设备 ID、缩放系数、MPS 工具和已发布的 CDI hook。 | +| Pod 无法匹配 GPU | 检查 Pod 与节点策略、算力和显存余量、SM chunk 和现有 GPU 集合。 | +| 时间片设置或回读失败 | 检查目标 GPU、驱动和基础时间片。插件会在排空后尝试启动两个同卡 server,再上报该策略。 | +| 资源回收等待客户端 | 等待 CUDA 客户端退出。HAMi 依次删除 namespace、分区和最后一个无引用的 server。 | +| daemon 丢失且仍有存活分配 | 排查 daemon 后通过 Kubernetes 重建受影响的工作负载。在 Pod 存活期间保留分配状态。 | + +容器仅在 `/run/hami-mps` 看到自己的 namespace socket 目录。算力设置对通过该 socket 接入的 CUDA 进程生效。MPS 不提供显存带宽或 MIG 级别的故障隔离。普通容器重启保留原分配。若仍有存活工作负载,管理器恢复依赖原 daemon 及其资源。 + +分配流程、恢复条件和真机数据见 [MPS v3 后端设计](../../developers/mps-v3-design.md)。 diff --git a/sidebars.js b/sidebars.js index 3487c0e9f..bec940759 100644 --- a/sidebars.js +++ b/sidebars.js @@ -91,6 +91,7 @@ module.exports = { items: [ "userguide/nvidia-device/dynamic-resource-allocation", "userguide/nvidia-device/dynamic-mig-support", + "userguide/nvidia-device/mps-v3", "userguide/nvidia-device/scheduling-policy", "userguide/nvidia-device/specify-device-memory-usage", "userguide/nvidia-device/specify-device-core-usage", @@ -389,6 +390,7 @@ module.exports = { "developers/scheduling", "developers/hami-core-design", "developers/dynamic-mig", + "developers/mps-v3-design", "developers/gpu-topology-scheduling", "developers/kunlunxin-topology", "developers/profiling-scheduler", From d2e917159208441b5ffccc923dd02b68bbf1e31e Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Thu, 12 Nov 2026 15:29:03 +0800 Subject: [PATCH 2/7] docs(nvidia): clarify MPS v3 architecture and allocation flow Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 71 ++++++++++++++----- docs/userguide/nvidia-device/mps-v3.md | 2 +- .../current/developers/mps-v3-design.md | 71 ++++++++++++++----- .../current/userguide/nvidia-device/mps-v3.md | 2 +- 4 files changed, 112 insertions(+), 34 deletions(-) diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md index 2e40c359f..f33d96a0a 100644 --- a/docs/developers/mps-v3-design.md +++ b/docs/developers/mps-v3-design.md @@ -5,6 +5,39 @@ sidebar_label: MPS v3 design HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 or later and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples. +## Architecture + +```mermaid +flowchart TB + subgraph HAMi["HAMi internals"] + direction LR + S["Scheduler"] --> D["Device plugin"] + D --> M["MPS manager"] + M <--> ST["Node state"] + D --> G["Physical GPU CDI"] + M --> A["Allocation CDI"] + end + subgraph External["External components"] + direction LR + N["MPS daemon"] + R["Container runtime"] + R --> W["CUDA application"] + W -->|namespace socket| N + end + M -->|MPS control CLI| N + A --> R + G --> R + D -.->|CDI device ID| R + style HAMi fill:#eaf2ff,stroke:#4073ad,color:#172e4a + style External fill:#e8f5ee,stroke:#3f805d,color:#183c2a + classDef internal fill:#f7faff,stroke:#4073ad,color:#172e4a + classDef external fill:#f5fcf7,stroke:#3f805d,color:#183c2a + class S,D,M,ST,A internal + class N,G,R,W external +``` + +The scheduler matches the node strategy and accounts for each GPU. The device plugin reads allocation annotations, and the MPS manager checks resource ownership and records allocation intent and completion on the node. HAMi generates the allocation CDI with a read-only socket mount and environment variables for routing and memory. The container runtime applies its mount and environment edits, and the CUDA client connects to the NVIDIA MPS namespace through the mounted socket. The device plugin uses NVIDIA Container Toolkit libraries to generate and publish the physical GPU CDI. The runtime invokes the memory hook specified in that CDI. + ## Scheduling contract The plugin reports `mps-v3`, the configured compute strategy, SM geometry, and allocated GPU sets. The scheduler matches `hami.io/mps-compute-mode` against the reported strategy, with `namespace-cap` as the Pod default. Both strategies use the existing per-GPU 100-unit compute budget and quota accounting. When core parsing returns zero, the backend charges 100 units for a percentage or timeslice request. @@ -20,6 +53,28 @@ Static partition accounting uses the base SM count. The scheduler validates each ## Allocation and runtime +### Allocation flow + +```mermaid +flowchart TD + V["Validate request"] --> I["Record intent"] + I --> B{"Compute strategy"} + B -->|server-timeslice| T["Create weighted server"] + B -->|namespace-cap| C["Get shared server"] + T --> TN["Create namespace"] + C --> CN["Configure namespace"] + TN --> Q["Verify allocation"] + CN --> Q + Q --> P["Publish CDI and save state"] + P --> A["Return Allocate response"] + T -.->|Operation failed| X["Roll back"] + C -.-> X + TN -.-> X + CN -.-> X + Q -.-> X + P -.-> X +``` + 1. Resolve the GPU set and check the allocation's strategy, memory, and compute settings. Check that any server to be reused has recorded ownership. 2. Persist the allocation intent with stable resource names before creating any resources. 3. Create or reuse the server. Set the weight for a new timeslice server before creating the namespace. For static SM requests, create each GPU's partition and bind it to the namespace using a separate `namespace set --device= --sm-partition=` operation. @@ -41,19 +96,3 @@ Health and recovery use individual `get` fields for configuration and use r620 C Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. If a plugin container restart also stops the daemon, affected workloads require recovery. Reclamation deletes the namespace without force, deletes its partitions, deletes the server once it has no remaining allocations, then removes the allocation's CDI entry and state. r620 refuses namespace deletion while clients are attached. Failed cleanup retains its record for retry. Normal container restarts keep their allocation because kubelet can reuse its Allocate response. - -## Verification on A10 - -The following results were observed on driver 620.70.06. - -| Check | Observed result | -| --- | --- | -| Two servers configured at 25% and 75% with a long base timeslice | Both settings read back and both servers ran on the same GPU. | -| Continuously queued CUDA Graph workload | Median GPU context residence was 12.9 ms and 38.4 ms. Throughput was approximately 1,024 and 2,816–3,072 kernels/s, respectively. | -| Synchronous short kernels | The two workloads had nearly equal GPU context residence and throughput. The remaining server returned to its single-workload throughput after competitors stopped. | -| SM borrowing with 32 base SMs | The borrower executed across 64 SMs with the lender idle, 40 after the lender connected, and 64 after it exited. The reported base count stayed at 32. | -| Static SM binding | A namespace assigned 16 SMs used that partition even when the client requested a different 4-SM partition through its environment. | -| Toolkit 1.20.1 CDI hook | Two separate container starts saw a 1 GiB limit and returned OOM for an oversized allocation, including after changing the in-process environment. | -| Lifecycle | Validated manager reattachment, policy changes after draining, state preservation after daemon loss, and refusal to delete a namespace with active clients. | - -These measurements characterize the tested GPU and workload. Physical multi-GPU and Kubernetes end-to-end coverage remain separate rollout checks. The Kubernetes installation on the test host used Toolkit 1.17.8. The CDI memory test used a separate Toolkit 1.20.1 installation. diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index 3c84f74b9..3d18cb7ed 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -120,4 +120,4 @@ HAMi binds each partition to the namespace by GPU UUID. The namespace uses an ac Each container receives a read-only mount of its namespace socket directory at `/run/hami-mps`. Compute settings apply to CUDA processes that attach through that socket. The MPS policies do not isolate memory bandwidth or provide MIG-style fault containment. Container restarts retain their allocations. Manager recovery requires the daemon and its resources to be available for live workloads. -See the [MPS v3 backend design](../../developers/mps-v3-design.md) for allocation, recovery, and hardware verification details. +See the [MPS v3 backend design](../../developers/mps-v3-design.md) for allocation and recovery details. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md index 39272025f..fbc8fdbf1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -5,6 +5,39 @@ sidebar_label: MPS v3 设计 HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限制 GPU 显存,并通过 MPS v3 共享算力。后端以 r620 为最低驱动版本,启动时检查专用 daemon 的命令接口。部署和工作负载示例见[用户指南](../userguide/nvidia-device/mps-v3.md)。 +## 架构 + +```mermaid +flowchart TB + subgraph HAMi["HAMi 内部"] + direction LR + S["调度器"] --> D["设备插件"] + D --> M["MPS 管理器"] + M <--> ST["节点状态"] + D --> G["物理 GPU CDI"] + M --> A["分配 CDI"] + end + subgraph External["外部组件"] + direction LR + N["MPS daemon"] + R["容器运行时"] + R --> W["CUDA 应用"] + W -->|namespace socket| N + end + M -->|MPS control CLI| N + A --> R + G --> R + D -.->|CDI device ID| R + style HAMi fill:#eaf2ff,stroke:#4073ad,color:#172e4a + style External fill:#e8f5ee,stroke:#3f805d,color:#183c2a + classDef internal fill:#f7faff,stroke:#4073ad,color:#172e4a + classDef external fill:#f5fcf7,stroke:#3f805d,color:#183c2a + class S,D,M,ST,A internal + class N,G,R,W external +``` + +调度器匹配节点策略并核算每卡额度。设备插件读取分配注解,由 MPS 管理器验证资源归属,并在节点上记录分配意图和完成状态。HAMi 生成每次分配的专属 CDI,提供只读 socket 挂载、路由和显存环境变量。容器运行时应用其中的挂载和环境变量,CUDA 客户端通过挂载的 socket 连接 NVIDIA MPS namespace。设备插件调用 NVIDIA Container Toolkit 的库生成并发布物理 GPU CDI。该配置包含显存 hook,由容器运行时调用。 + ## 调度契约 设备插件上报 `mps-v3`、节点算力策略、SM 几何信息和已分配的 GPU 集合。调度器按 `hami.io/mps-compute-mode` 匹配节点策略,Pod 缺省为 `namespace-cap`。两种策略沿用每卡 100 单位的算力预算和现有配额。百分比或时间片请求的算力解析结果为 0 时,按 100 单位记账。 @@ -20,6 +53,28 @@ HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限 ## 分配与容器启动 +### 分配流程 + +```mermaid +flowchart TD + V["校验请求"] --> I["记录意图"] + I --> B{"算力策略"} + B -->|server-timeslice| T["创建加权 server"] + B -->|namespace-cap| C["获取共享 server"] + T --> TN["创建 namespace"] + C --> CN["配置 namespace"] + TN --> Q["回读校验"] + CN --> Q + Q --> P["发布 CDI 并落盘"] + P --> A["返回 Allocate 响应"] + T -.->|操作失败| X["回滚"] + C -.-> X + TN -.-> X + CN -.-> X + Q -.-> X + P -.-> X +``` + 1. 校验 GPU 集合、策略、显存和算力设置。复用 server 前验证其归属。 2. 在创建资源前将稳定的资源名称和分配意图写入节点状态。 3. 创建或复用 server。新时间片 server 先设置权重,再创建 namespace。静态 SM 请求先按卡创建分区,再逐卡单独执行 `namespace set --device= --sm-partition=`。 @@ -41,19 +96,3 @@ HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限 恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。整个插件容器重启若同时结束了 daemon,则需要恢复受影响的工作负载。 回收依次执行非强制删除 namespace、删除分区、删除无引用 server,再移除 CDI 和状态。r620 会拒绝删除仍有客户端的 namespace,失败记录保留供重试。普通容器重启保留原分配,以便 kubelet 复用 Allocate 响应。 - -## A10 真机验证 - -以下结果来自驱动 620.70.06。 - -| 项目 | 观察结果 | -| --- | --- | -| 两个 server 配置 25% / 75% 长时间片 | 权重回读成功,两个 server 在同一张卡上运行。 | -| 持续排队的 CUDA Graph | GPU 上下文驻留中位数约为 12.9 / 38.4 ms,吞吐约为 1,024 / 2,816–3,072 kernel/s。 | -| 同步短 kernel | 两个工作负载的 GPU 上下文驻留与吞吐均接近。其他 server 停止执行后,剩余负载恢复到独自运行时的吞吐。 | -| 基础额度 32 SM 的借用 | 借出方空闲时,借入方实际覆盖 64 个 SM。借出方接入后降为 40,退出后恢复到 64。CUDA 报告的基础值仍为 32。 | -| 服务端静态绑定 | namespace 绑定 16 SM 后,即使客户端通过环境变量请求另一块 4 SM 分区,仍获得 16 SM。 | -| Toolkit 1.20.1 CDI hook | 两次容器启动均看到 1 GiB 上限。进程内修改环境变量后,超额申请仍返回 OOM。 | -| 生命周期 | 验证管理器重连、排空后切换、daemon 失联后保留状态,以及驱动阻止删除有活跃客户端的 namespace。 | - -实测值取决于 GPU 和负载。物理多卡和 Kubernetes 端到端仍待验证。测试机的集群使用 Toolkit 1.17.8,CDI 显存实验使用独立的 Toolkit 1.20.1。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index fe4f60b9e..b9b77589c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -120,4 +120,4 @@ HAMi 按 GPU UUID 将分区绑定到 namespace,并将 namespace 的活跃线 容器仅在 `/run/hami-mps` 看到自己的 namespace socket 目录。算力设置对通过该 socket 接入的 CUDA 进程生效。MPS 不提供显存带宽或 MIG 级别的故障隔离。普通容器重启保留原分配。若仍有存活工作负载,管理器恢复依赖原 daemon 及其资源。 -分配流程、恢复条件和真机数据见 [MPS v3 后端设计](../../developers/mps-v3-design.md)。 +分配流程和恢复条件见 [MPS v3 后端设计](../../developers/mps-v3-design.md)。 From c0c9ae18716af94b066653a72d8e35656a0ab138 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Thu, 12 Nov 2026 18:53:30 +0800 Subject: [PATCH 3/7] docs(nvidia): update MPS v3 health checks and requirements Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 6 +++--- docs/userguide/nvidia-device/mps-v3.md | 2 +- .../current/developers/mps-v3-design.md | 6 +++--- .../current/userguide/nvidia-device/mps-v3.md | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md index f33d96a0a..ec60e950c 100644 --- a/docs/developers/mps-v3-design.md +++ b/docs/developers/mps-v3-design.md @@ -49,7 +49,7 @@ The plugin reports `mps-v3`, the configured compute strategy, SM geometry, and a Each container uses the same memory and compute allocation on every requested GPU. Identical GPU sets can share a server in `namespace-cap` and can have independent servers in `server-timeslice`. Partial overlaps are rejected by scheduling and checked again by the manager. Percentage and static SM provisioning cannot share a GPU. -Static partition accounting uses the base SM count. The scheduler validates each GPU's chunk geometry and charges `ceil(base SMs / total SMs × 100)`. If lending or borrowing is enabled, the base must consist of whole chunks. The finite maximum must cover the base and fit the GPU. The driver's runtime borrowing decisions do not change scheduler reservations. +Static partition accounting uses the base SM count. Before creating static partitions, the manager checks the driver-reported total SMs and chunks against the scheduler geometry. The scheduler charges `ceil(base SMs / total SMs × 100)`. If lending or borrowing is enabled, the base must consist of whole chunks. The finite maximum must cover the base and fit the GPU. The driver's runtime borrowing decisions do not change scheduler reservations. ## Allocation and runtime @@ -89,9 +89,9 @@ Compute routing uses the namespace socket. A future cgroup routing integration r The manager takes an exclusive lock on the dedicated state directory. It records the state version, Pod and container identity, GPU set, limits, SM options, resource names, and preparation status in `allocations.json`. It uses the node's Pod list to reconcile these records with the daemon. -On a drained daemon, the manager sets `compute-policy` and verifies the result. A drained `server-timeslice` node probes two concurrently ready servers for the configured GPU set. If yield was previously set and the desired value is omitted, the manager restarts the drained daemon to restore the driver default. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) +On a drained daemon, the manager sets `compute-policy` and verifies the result. For `server-timeslice`, the plugin checks each GPU through NVML and makes only GPUs in `DEFAULT` compute mode eligible. The manager validates the actual GPU set when creating a workload server. If yield was previously set and the desired value is omitted, the manager restarts the drained daemon to restore the driver default. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) -Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Failed daemon or allocation health checks mark devices unhealthy and block new allocations. +Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Daemon or global policy failures mark all GPUs unhealthy. An allocation health check failure affects only the GPUs in that allocation. Read-only health checks run outside the allocation locks. Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. If a plugin container restart also stops the daemon, affected workloads require recovery. diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index 3d18cb7ed..ff77dd755 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -114,7 +114,7 @@ HAMi binds each partition to the namespace by GPU UUID. The namespace uses an ac | --- | --- | | The node rejects MPS v3 during startup | Check driver r620 or later, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. | | A Pod cannot find a compatible GPU | Check the Pod and node strategies, available core and memory budget, the SM chunk geometry, and existing GPU sets. | -| Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. The plugin probes two concurrent servers on a drained node before advertising the strategy. | +| Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. Each GPU must use `DEFAULT` compute mode for `server-timeslice`. | | Cleanup is waiting for clients | Let clients exit. HAMi removes the namespace first, then the partition and the last unused server. | | The daemon is lost with live allocations | Investigate the daemon and restart affected workloads through Kubernetes. Preserve the allocation state while the Pods are live. | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md index fbc8fdbf1..d24f2f3e5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -49,7 +49,7 @@ flowchart TB 同一容器在每张卡上使用相同的显存和算力额度。相同 GPU 集合在 `namespace-cap` 下可共用 server,在 `server-timeslice` 下可运行多个独立 server。调度器和管理器均拒绝部分重叠的集合。同一 GPU 不能混用静态 SM 分区和 namespace 百分比。 -静态分区以基础 SM 数核算。调度器校验每卡的 chunk 几何,并按 `ceil(基础 SM 数 / 总 SM 数 × 100)` 记账。开启借出或借入时,基础额度须由整 chunk 组成,有限峰值须覆盖基础额度并符合每卡容量。驱动在运行时决定借用量,调度预留保持基础额度。 +静态分区以基础 SM 数核算。创建静态分区前,管理器核对驱动报告的总 SM 数和 chunk 数。调度器按 `ceil(基础 SM 数 / 总 SM 数 × 100)` 记账。开启借出或借入时,基础额度须由整 chunk 组成,有限峰值须覆盖基础额度并符合每卡容量。驱动在运行时决定借用量,调度预留保持基础额度。 ## 分配与容器启动 @@ -89,9 +89,9 @@ flowchart TD 管理器对专用状态目录加独占锁,在 `allocations.json` 记录版本、Pod 与容器身份、GPU 集合、额度、SM 选项、资源名称和准备状态。启动时根据权威节点 Pod 列表和 daemon 状态恢复分配。 -排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点用配置的 GPU 集合验证两个 server 能同时就绪。如果要恢复为默认 yield,管理器会在无 server 时重启 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) +排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点通过 NVML 逐卡检查 compute mode,仅允许使用 `DEFAULT` 模式的 GPU。实际 GPU 集合在创建工作负载的 server 时验证。如果要恢复为默认 yield,管理器会在无 server 时重启 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) -健康检查和恢复优先使用 `get` 字段回读配置。资源是否存在、静态分区的具体设置通过 r620 CSV inventory 核对。620.70.06 的 `sm-partition list --all` 在状态列前还有一列 contract timeout。未知格式、查询超时或断连会停止当前操作并保留记录。daemon 或分配健康检查失败时,插件会上报设备不健康并阻止新分配。 +健康检查和恢复优先使用 `get` 字段回读配置。资源是否存在、静态分区的具体设置通过 r620 CSV inventory 核对。620.70.06 的 `sm-partition list --all` 在状态列前还有一列 contract timeout。未知格式、查询超时或断连会停止当前操作并保留记录。daemon 或全局策略异常影响整节点,单个分配的健康异常仅影响其 GPU。只读健康检查在分配锁外执行。 恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。整个插件容器重启若同时结束了 daemon,则需要恢复受影响的工作负载。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index b9b77589c..f6a599754 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -114,7 +114,7 @@ HAMi 按 GPU UUID 将分区绑定到 namespace,并将 namespace 的活跃线 | --- | --- | | MPS v3 节点启动失败 | 检查 r620 驱动、cgroup v2、非 MIG GPU、UUID 设备 ID、缩放系数、MPS 工具和已发布的 CDI hook。 | | Pod 无法匹配 GPU | 检查 Pod 与节点策略、算力和显存余量、SM chunk 和现有 GPU 集合。 | -| 时间片设置或回读失败 | 检查目标 GPU、驱动和基础时间片。插件会在排空后尝试启动两个同卡 server,再上报该策略。 | +| 时间片设置或回读失败 | 检查目标 GPU、驱动和基础时间片。`server-timeslice` 要求 GPU 使用 `DEFAULT` compute mode。 | | 资源回收等待客户端 | 等待 CUDA 客户端退出。HAMi 依次删除 namespace、分区和最后一个无引用的 server。 | | daemon 丢失且仍有存活分配 | 排查 daemon 后通过 Kubernetes 重建受影响的工作负载。在 Pod 存活期间保留分配状态。 | From 6587da0d49c2c18d93024793c3ba2fc99d88a279 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 16 Nov 2026 11:07:50 +0800 Subject: [PATCH 4/7] docs(nvidia): clarify upstream MPS compatibility Signed-off-by: Jifei Wang --- docs/userguide/nvidia-device/mps-v3.md | 6 ++++++ .../current/userguide/nvidia-device/mps-v3.md | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index ff77dd755..499327c95 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -5,6 +5,12 @@ sidebar_label: MPS v3 The `mps-v3` backend uses the NVIDIA Container Toolkit CDI hook to apply each container's GPU memory limit to its cgroup. CUDA processes connect to their assigned MPS namespace for compute sharing. This backend uses the existing HAMi GPU, memory, and core resources. +## NVIDIA device plugin compatibility + +HAMi enables r620 MPS v3 with `operatingmode: mps-v3`. It supports per-container GPU memory limits through CDI and offers namespace caps, static SM partitions with optional borrowing, and weighted server timeslices. + +The upstream NVIDIA device plugin's experimental `sharing.mps` uses Legacy MPS to expose equal-sized GPU replicas. That setting is unavailable in HAMi, where the node operating mode for MPS is `mps-v3`. [NVIDIA device plugin MPS documentation](https://github.com/NVIDIA/k8s-device-plugin/blob/v0.20.1/README.md#with-cuda-mps) + ## Prepare the node - Use Linux with cgroup v2, NVIDIA driver **r620 or later**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index f6a599754..34b33594e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -5,6 +5,12 @@ sidebar_label: MPS v3 `mps-v3` 后端通过 NVIDIA Container Toolkit 的 CDI hook 为每个容器的 cgroup 设置显存上限。CUDA 进程连接分配给容器的 MPS namespace,按节点配置的策略共享算力。GPU 数量、显存和算力继续使用现有的 HAMi 资源名。 +## 与 NVIDIA device plugin 的关系 + +HAMi 通过 `operatingmode: mps-v3` 启用 r620 MPS v3,用 CDI 设置容器显存上限,并提供 namespace 算力上限、可选借用的静态 SM 分区和 server 加权时间片。 + +NVIDIA 上游 device plugin 的实验性 `sharing.mps` 使用旧版 MPS,将 GPU 暴露为等额副本。HAMi 不支持该配置,启用 MPS 时使用节点模式 `mps-v3`。[NVIDIA device plugin MPS 文档](https://github.com/NVIDIA/k8s-device-plugin/blob/v0.20.1/README.md#with-cuda-mps) + ## 准备节点 - 使用 cgroup v2、**r620 或更新的 NVIDIA 驱动**和 **NVIDIA Container Toolkit 1.20.1 或更新版本**。启用 `apply-cuda-memory-limits` hook。应用镜像应使用与节点驱动兼容的 CUDA 运行时。 From 3d3e875d5e08531a42ce4087bd61717b693478f7 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 16 Nov 2026 15:15:24 +0800 Subject: [PATCH 5/7] docs(nvidia): address MPS v3 guide review Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 1 + docs/userguide/nvidia-device/mps-v3.md | 3 +++ .../current/developers/mps-v3-design.md | 1 + .../current/userguide/nvidia-device/mps-v3.md | 3 +++ 4 files changed, 8 insertions(+) diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md index ec60e950c..f21138283 100644 --- a/docs/developers/mps-v3-design.md +++ b/docs/developers/mps-v3-design.md @@ -1,5 +1,6 @@ --- title: NVIDIA MPS v3 backend design +description: Architecture, GPU scheduling, allocation, recovery, and cleanup for the HAMi MPS v3 backend. sidebar_label: MPS v3 design --- diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index 499327c95..05b0cf0c9 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -1,10 +1,13 @@ --- title: Share NVIDIA GPUs with MPS v3 +description: Configure MPS v3 in HAMi and request GPU memory, compute caps, static SM partitions, or weighted timeslices. sidebar_label: MPS v3 --- The `mps-v3` backend uses the NVIDIA Container Toolkit CDI hook to apply each container's GPU memory limit to its cgroup. CUDA processes connect to their assigned MPS namespace for compute sharing. This backend uses the existing HAMi GPU, memory, and core resources. +The HAMi implementation is tracked in [HAMi PR #3443](https://github.com/Project-HAMi/HAMi/pull/3443). + ## NVIDIA device plugin compatibility HAMi enables r620 MPS v3 with `operatingmode: mps-v3`. It supports per-container GPU memory limits through CDI and offers namespace caps, static SM partitions with optional borrowing, and weighted server timeslices. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md index d24f2f3e5..ec5f4b0fb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -1,5 +1,6 @@ --- title: NVIDIA MPS v3 后端设计 +description: 介绍 HAMi MPS v3 后端的架构、GPU 调度、分配、恢复和回收。 sidebar_label: MPS v3 设计 --- diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index 34b33594e..d5cc54015 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -1,10 +1,13 @@ --- title: 使用 MPS v3 共享 NVIDIA GPU +description: 介绍如何在 HAMi 中配置 MPS v3,并申请显存、算力上限、静态 SM 分区和加权时间片。 sidebar_label: MPS v3 --- `mps-v3` 后端通过 NVIDIA Container Toolkit 的 CDI hook 为每个容器的 cgroup 设置显存上限。CUDA 进程连接分配给容器的 MPS namespace,按节点配置的策略共享算力。GPU 数量、显存和算力继续使用现有的 HAMi 资源名。 +HAMi 实现见 [HAMi PR #3443](https://github.com/Project-HAMi/HAMi/pull/3443)。 + ## 与 NVIDIA device plugin 的关系 HAMi 通过 `operatingmode: mps-v3` 启用 r620 MPS v3,用 CDI 设置容器显存上限,并提供 namespace 算力上限、可选借用的静态 SM 分区和 server 加权时间片。 From dbf3aa2dfdc6a84b9016d24ad3359eaeb2ef49be Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 16 Nov 2026 17:13:31 +0800 Subject: [PATCH 6/7] docs(nvidia): document r620 and unified SM annotation Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 2 +- docs/userguide/nvidia-device/mps-v3.md | 8 ++++---- .../current/developers/mps-v3-design.md | 2 +- .../current/userguide/nvidia-device/mps-v3.md | 6 +++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md index f21138283..3322a6617 100644 --- a/docs/developers/mps-v3-design.md +++ b/docs/developers/mps-v3-design.md @@ -4,7 +4,7 @@ description: Architecture, GPU scheduling, allocation, recovery, and cleanup for sidebar_label: MPS v3 design --- -HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 or later and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples. +HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples. ## Architecture diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index 05b0cf0c9..7d54b2234 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -16,7 +16,7 @@ The upstream NVIDIA device plugin's experimental `sharing.mps` uses Legacy MPS t ## Prepare the node -- Use Linux with cgroup v2, NVIDIA driver **r620 or later**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver. +- Use Linux with cgroup v2, NVIDIA driver **r620**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver. - Disable MIG before enrolling the node. Configure a CDI-capable container runtime and UUID device IDs. Set device memory and core scaling to `1` on MPS v3 nodes. - Make the driver-matched `nvidia-cuda-mps-control` and its libraries available inside the privileged device plugin, which uses the host PID namespace. Set `devicePlugin.mpsV3.binary` to the path visible inside the plugin. The chart's host `/usr/bin` mount is visible at `/usrbin`. With GPU Operator, locate the binary in the configured driver installation. - Verify that the published CDI device for each GPU includes the memory hook and that the hook path is executable on the host. HAMi checks the published spec before registering devices. The Toolkit and driver manage the memory controller. [NVIDIA memory partitioning](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) @@ -101,12 +101,12 @@ For a `namespace-cap` Pod, add the following annotations and omit `gpucores` if metadata: annotations: nvidia.com/vgpu-mode: mps-v3 - nvidia.com/mps-sm-count: "16" + hami.io/mps-sm-count: "16" hami.io/mps-sm-shareable: "true" hami.io/mps-sm-max-chunks: "16" ``` -- `nvidia.com/mps-sm-count` applies to each GPU of every GPU container in the Pod. An Ampere discrete GPU uses 2 SMs per chunk, so the example reserves eight chunks. Hopper and newer discrete GPUs use 8 SMs per chunk. Integrated GPUs use 2 SMs per chunk. A fixed partition can also use the device's remainder if it yields the exact requested count. +- `hami.io/mps-sm-count` applies to each GPU of every GPU container in the Pod. An Ampere discrete GPU uses 2 SMs per chunk, so the example reserves eight chunks. Hopper and newer discrete GPUs use 8 SMs per chunk. Integrated GPUs use 2 SMs per chunk. A fixed partition can also use the device's remainder if it yields the exact requested count. - `hami.io/mps-sm-shareable` allows the driver to lend the partition's idle SMs while it has no attached clients. Its default is `false`. - `hami.io/mps-sm-max-chunks` caps the **total** chunks the partition may use, including its base reservation. The default `0` disables borrowing. A positive integer must be between the base and total chunks on every allocated GPU. Use `unlimited` to let the driver use available capacity without a configured cap. - Borrowing or lending requires a base reservation made of whole chunks. The scheduler and quota account for the base reservation as `ceil(requested SMs / total SMs × 100)`. An explicit `gpucores` value must cover this charge. The driver controls how many additional SMs can be used at runtime. @@ -121,7 +121,7 @@ HAMi binds each partition to the namespace by GPU UUID. The namespace uses an ac | Symptom | What to check | | --- | --- | -| The node rejects MPS v3 during startup | Check driver r620 or later, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. | +| The node rejects MPS v3 during startup | Check driver r620, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. | | A Pod cannot find a compatible GPU | Check the Pod and node strategies, available core and memory budget, the SM chunk geometry, and existing GPU sets. | | Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. Each GPU must use `DEFAULT` compute mode for `server-timeslice`. | | Cleanup is waiting for clients | Let clients exit. HAMi removes the namespace first, then the partition and the last unused server. | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md index ec5f4b0fb..842bc8ea8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -4,7 +4,7 @@ description: 介绍 HAMi MPS v3 后端的架构、GPU 调度、分配、恢复 sidebar_label: MPS v3 设计 --- -HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限制 GPU 显存,并通过 MPS v3 共享算力。后端以 r620 为最低驱动版本,启动时检查专用 daemon 的命令接口。部署和工作负载示例见[用户指南](../userguide/nvidia-device/mps-v3.md)。 +HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限制 GPU 显存,并通过 MPS v3 共享算力。后端仅支持 r620 驱动分支,启动时检查专用 daemon 的命令接口。部署和工作负载示例见[用户指南](../userguide/nvidia-device/mps-v3.md)。 ## 架构 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index d5cc54015..66e1c4653 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -16,7 +16,7 @@ NVIDIA 上游 device plugin 的实验性 `sharing.mps` 使用旧版 MPS,将 GP ## 准备节点 -- 使用 cgroup v2、**r620 或更新的 NVIDIA 驱动**和 **NVIDIA Container Toolkit 1.20.1 或更新版本**。启用 `apply-cuda-memory-limits` hook。应用镜像应使用与节点驱动兼容的 CUDA 运行时。 +- 使用 cgroup v2、**NVIDIA r620 驱动**和 **NVIDIA Container Toolkit 1.20.1 或更新版本**。启用 `apply-cuda-memory-limits` hook。应用镜像应使用与节点驱动兼容的 CUDA 运行时。 - 启用后端前先关闭 MIG。使用支持 CDI 的容器运行时,以 UUID 标识 GPU,并将节点的显存和算力缩放系数设为 `1`。 - 确认特权 device plugin 使用主机 PID 命名空间,并能访问与驱动配套的 `nvidia-cuda-mps-control` 和驱动库。Chart 将主机的 `/usr/bin` 挂到插件容器内的 `/usrbin`。使用 GPU Operator 时,请按驱动的实际安装位置配置二进制路径。 - 检查已经发布的物理 GPU CDI 设备包含显存 hook,且 hook 路径在主机上可执行。HAMi 在注册设备前会检查该配置。显存控制器由 Toolkit 和驱动管理。[NVIDIA 显存分区文档](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) @@ -101,12 +101,12 @@ spec: metadata: annotations: nvidia.com/vgpu-mode: mps-v3 - nvidia.com/mps-sm-count: "16" + hami.io/mps-sm-count: "16" hami.io/mps-sm-shareable: "true" hami.io/mps-sm-max-chunks: "16" ``` -- `nvidia.com/mps-sm-count` 适用于 Pod 内每个 GPU 容器申请的每张卡。Ampere 独立 GPU 每个 chunk 为 2 SM,因此示例预留 8 个 chunk。Hopper 及更新的独立 GPU 每个 chunk 为 8 SM,集成 GPU 为 2 SM。固定分区也可以使用该设备的余数 SM,前提是能精确组成请求值。 +- `hami.io/mps-sm-count` 适用于 Pod 内每个 GPU 容器申请的每张卡。Ampere 独立 GPU 每个 chunk 为 2 SM,因此示例预留 8 个 chunk。Hopper 及更新的独立 GPU 每个 chunk 为 8 SM,集成 GPU 为 2 SM。固定分区也可以使用该设备的余数 SM,前提是能精确组成请求值。 - `hami.io/mps-sm-shareable` 允许驱动在分区没有客户端时借出空闲 SM。默认 `false`。 - `hami.io/mps-sm-max-chunks` 是分区能使用的**总** chunk 数上限,包含基础额度。默认 `0` 表示禁止借入。正整数上限必须在每卡的基础 chunk 数与该卡总 chunk 数之间。`unlimited` 允许驱动使用可借容量。 - 启用借出或借入时,基础额度必须是整 chunk。调度和配额按 `ceil(基础 SM 数 / GPU 总 SM 数 × 100)` 记账。显式设置的 `gpucores` 必须覆盖该值。运行时实际借用量由驱动决定。 From 508da9f2129f76cd2910c989c89366cc2556de31 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 16 Nov 2026 17:35:05 +0800 Subject: [PATCH 7/7] docs(nvidia): deploy MPS v3 as a host service Signed-off-by: Jifei Wang --- docs/developers/mps-v3-design.md | 4 +-- docs/userguide/nvidia-device/mps-v3.md | 30 +++++++++++++++++++ .../current/developers/mps-v3-design.md | 4 +-- .../current/userguide/nvidia-device/mps-v3.md | 30 +++++++++++++++++++ 4 files changed, 64 insertions(+), 4 deletions(-) diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md index 3322a6617..ef599ce56 100644 --- a/docs/developers/mps-v3-design.md +++ b/docs/developers/mps-v3-design.md @@ -90,10 +90,10 @@ Compute routing uses the namespace socket. A future cgroup routing integration r The manager takes an exclusive lock on the dedicated state directory. It records the state version, Pod and container identity, GPU set, limits, SM options, resource names, and preparation status in `allocations.json`. It uses the node's Pod list to reconcile these records with the daemon. -On a drained daemon, the manager sets `compute-policy` and verifies the result. For `server-timeslice`, the plugin checks each GPU through NVML and makes only GPUs in `DEFAULT` compute mode eligible. The manager validates the actual GPU set when creating a workload server. If yield was previously set and the desired value is omitted, the manager restarts the drained daemon to restore the driver default. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) +On a drained daemon, the manager sets `compute-policy` and verifies the result. For `server-timeslice`, the plugin checks each GPU through NVML and makes only GPUs in `DEFAULT` compute mode eligible. The manager validates the actual GPU set when creating a workload server. Restoring the default yield requires the operator to drain the node and restart the host daemon. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Daemon or global policy failures mark all GPUs unhealthy. An allocation health check failure affects only the GPUs in that allocation. Read-only health checks run outside the allocation locks. -Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. If a plugin container restart also stops the daemon, affected workloads require recovery. +Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. The host service owns the daemon lifecycle. If the host daemon exits, affected workloads require recovery. Reclamation deletes the namespace without force, deletes its partitions, deletes the server once it has no remaining allocations, then removes the allocation's CDI entry and state. r620 refuses namespace deletion while clients are attached. Failed cleanup retains its record for retry. Normal container restarts keep their allocation because kubelet can reuse its Allocate response. diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md index 7d54b2234..40b9f5750 100644 --- a/docs/userguide/nvidia-device/mps-v3.md +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -26,6 +26,36 @@ stat -fc %T /sys/fs/cgroup nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv ``` +## Run the MPS daemon on the host + +Start a dedicated daemon on each MPS v3 node before enabling the device plugin. This example systemd unit uses the host binary path. The directories must match the Chart setting `devicePlugin.mpsV3.root`. + +```ini +[Unit] +Description=HAMi NVIDIA MPS v3 daemon + +[Service] +Type=simple +Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe +Environment=CUDA_MPS_LOG_DIRECTORY=/var/run/hami/mps-v3/log +ExecStartPre=/usr/bin/mkdir -p /var/run/hami/mps-v3/pipe /var/run/hami/mps-v3/log +ExecStart=/usr/bin/nvidia-cuda-mps-control -p 3 -f -m + +[Install] +WantedBy=multi-user.target +``` + +Save the unit as `/etc/systemd/system/hami-mps-v3.service` and run the following on the node. + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now hami-mps-v3.service +sudo env CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe \ + nvidia-cuda-mps-control -p 3 compute-policy get timeslice +``` + +The device plugin connects to the running daemon. Drain MPS workloads before stopping or restarting the host service. + ## Configure a node Select the node and choose its compute strategy in the Helm overrides. This example enables `server-timeslice` on one node. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md index 842bc8ea8..858450b41 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -90,10 +90,10 @@ flowchart TD 管理器对专用状态目录加独占锁,在 `allocations.json` 记录版本、Pod 与容器身份、GPU 集合、额度、SM 选项、资源名称和准备状态。启动时根据权威节点 Pod 列表和 daemon 状态恢复分配。 -排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点通过 NVML 逐卡检查 compute mode,仅允许使用 `DEFAULT` 模式的 GPU。实际 GPU 集合在创建工作负载的 server 时验证。如果要恢复为默认 yield,管理器会在无 server 时重启 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) +排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点通过 NVML 逐卡检查 compute mode,仅允许使用 `DEFAULT` 模式的 GPU。实际 GPU 集合在创建工作负载的 server 时验证。恢复默认 yield 时,运维需先排空节点并重启宿主机 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) 健康检查和恢复优先使用 `get` 字段回读配置。资源是否存在、静态分区的具体设置通过 r620 CSV inventory 核对。620.70.06 的 `sm-partition list --all` 在状态列前还有一列 contract timeout。未知格式、查询超时或断连会停止当前操作并保留记录。daemon 或全局策略异常影响整节点,单个分配的健康异常仅影响其 GPU。只读健康检查在分配锁外执行。 -恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。整个插件容器重启若同时结束了 daemon,则需要恢复受影响的工作负载。 +恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。daemon 由宿主机服务托管,插件重启后重新连接。如果宿主机 daemon 退出,则需要恢复受影响的工作负载。 回收依次执行非强制删除 namespace、删除分区、删除无引用 server,再移除 CDI 和状态。r620 会拒绝删除仍有客户端的 namespace,失败记录保留供重试。普通容器重启保留原分配,以便 kubelet 复用 Allocate 响应。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md index 66e1c4653..6c237427b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -26,6 +26,36 @@ stat -fc %T /sys/fs/cgroup nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv ``` +## 在宿主机运行 MPS daemon + +在启用 MPS v3 的节点上先启动专用 daemon,再启用设备插件。下面是 systemd unit 示例。二进制使用宿主机路径,目录须与 Chart 的 `devicePlugin.mpsV3.root` 一致。 + +```ini +[Unit] +Description=HAMi NVIDIA MPS v3 daemon + +[Service] +Type=simple +Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe +Environment=CUDA_MPS_LOG_DIRECTORY=/var/run/hami/mps-v3/log +ExecStartPre=/usr/bin/mkdir -p /var/run/hami/mps-v3/pipe /var/run/hami/mps-v3/log +ExecStart=/usr/bin/nvidia-cuda-mps-control -p 3 -f -m + +[Install] +WantedBy=multi-user.target +``` + +保存为 `/etc/systemd/system/hami-mps-v3.service` 后,在节点上执行: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now hami-mps-v3.service +sudo env CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe \ + nvidia-cuda-mps-control -p 3 compute-policy get timeslice +``` + +设备插件只连接已有 daemon。停止或重启节点服务前,先排空 MPS 工作负载。 + ## 配置节点 以下 Helm 配置在 `gpu-node-01` 上启用 `server-timeslice`。按实际节点名称和驱动路径修改。