diff --git a/docs/developers/mps-v3-design.md b/docs/developers/mps-v3-design.md new file mode 100644 index 000000000..ef599ce56 --- /dev/null +++ b/docs/developers/mps-v3-design.md @@ -0,0 +1,99 @@ +--- +title: NVIDIA MPS v3 backend design +description: Architecture, GPU scheduling, allocation, recovery, and cleanup for the HAMi MPS v3 backend. +sidebar_label: MPS v3 design +--- + +HAMi uses the NVIDIA Container Toolkit's `apply-cuda-memory-limits` CDI hook for GPU memory limits and MPS v3 for compute sharing. The backend requires driver r620 and validates its command interface against a dedicated daemon. See the [user guide](../userguide/nvidia-device/mps-v3.md) for prerequisites and workload examples. + +## Architecture + +```mermaid +flowchart TB + subgraph HAMi["HAMi internals"] + direction LR + S["Scheduler"] --> D["Device plugin"] + D --> M["MPS manager"] + M <--> ST["Node state"] + D --> G["Physical GPU CDI"] + M --> A["Allocation CDI"] + end + subgraph External["External components"] + direction LR + N["MPS daemon"] + R["Container runtime"] + R --> W["CUDA application"] + W -->|namespace socket| N + end + M -->|MPS control CLI| N + A --> R + G --> R + D -.->|CDI device ID| R + style HAMi fill:#eaf2ff,stroke:#4073ad,color:#172e4a + style External fill:#e8f5ee,stroke:#3f805d,color:#183c2a + classDef internal fill:#f7faff,stroke:#4073ad,color:#172e4a + classDef external fill:#f5fcf7,stroke:#3f805d,color:#183c2a + class S,D,M,ST,A internal + class N,G,R,W external +``` + +The scheduler matches the node strategy and accounts for each GPU. The device plugin reads allocation annotations, and the MPS manager checks resource ownership and records allocation intent and completion on the node. HAMi generates the allocation CDI with a read-only socket mount and environment variables for routing and memory. The container runtime applies its mount and environment edits, and the CUDA client connects to the NVIDIA MPS namespace through the mounted socket. The device plugin uses NVIDIA Container Toolkit libraries to generate and publish the physical GPU CDI. The runtime invokes the memory hook specified in that CDI. + +## Scheduling contract + +The plugin reports `mps-v3`, the configured compute strategy, SM geometry, and allocated GPU sets. The scheduler matches `hami.io/mps-compute-mode` against the reported strategy, with `namespace-cap` as the Pod default. Both strategies use the existing per-GPU 100-unit compute budget and quota accounting. When core parsing returns zero, the backend charges 100 units for a percentage or timeslice request. + +| Strategy | Server identity | Runtime provisioning | +| --- | --- | --- | +| `namespace-cap` | Sorted GPU UUID set | One namespace per Pod UID and container name. HAMi sets its active-thread percentage or assigns a static SM partition on every GPU. | +| `server-timeslice` | Pod UID, container name, and sorted GPU UUID set | One server per GPU container. HAMi programs and reads back `timeslice-pct` before the container starts and sets the namespace percentage to 100. | + +Each container uses the same memory and compute allocation on every requested GPU. Identical GPU sets can share a server in `namespace-cap` and can have independent servers in `server-timeslice`. Partial overlaps are rejected by scheduling and checked again by the manager. Percentage and static SM provisioning cannot share a GPU. + +Static partition accounting uses the base SM count. Before creating static partitions, the manager checks the driver-reported total SMs and chunks against the scheduler geometry. The scheduler charges `ceil(base SMs / total SMs × 100)`. If lending or borrowing is enabled, the base must consist of whole chunks. The finite maximum must cover the base and fit the GPU. The driver's runtime borrowing decisions do not change scheduler reservations. + +## Allocation and runtime + +### Allocation flow + +```mermaid +flowchart TD + V["Validate request"] --> I["Record intent"] + I --> B{"Compute strategy"} + B -->|server-timeslice| T["Create weighted server"] + B -->|namespace-cap| C["Get shared server"] + T --> TN["Create namespace"] + C --> CN["Configure namespace"] + TN --> Q["Verify allocation"] + CN --> Q + Q --> P["Publish CDI and save state"] + P --> A["Return Allocate response"] + T -.->|Operation failed| X["Roll back"] + C -.-> X + TN -.-> X + CN -.-> X + Q -.-> X + P -.-> X +``` + +1. Resolve the GPU set and check the allocation's strategy, memory, and compute settings. Check that any server to be reused has recorded ownership. +2. Persist the allocation intent with stable resource names before creating any resources. +3. Create or reuse the server. Set the weight for a new timeslice server before creating the namespace. For static SM requests, create each GPU's partition and bind it to the namespace using a separate `namespace set --device= --sm-partition=` operation. +4. Check the server's GPU set, namespace percentage, timeslice weight where applicable, and the socket. Read `sm-partitions` by GPU UUID and check the underlying partition configuration. Recovery checks base chunks, remainder, lending and borrowing parameters, and binding. Changes to effective SM capacity during borrowing are expected. +5. Publish the allocation's CDI entry and persist the completed state before returning the allocation. If any container in the request fails, roll back the resources newly created for that request. + +The allocation-specific CDI entry supplies the read-only namespace socket mount, `CUDA_MPS_PIPE_DIRECTORY`, and integer MiB values for `NVIDIA_GPU_MEMORY_REQUEST` and `NVIDIA_GPU_MEMORY_LIMIT`. The physical GPU's CDI entry invokes the Toolkit hook during `createRuntime`. The namespace mount exposes only the allocated namespace. Static partition selection is handled by the server, which takes precedence over a client-provided partition identifier. [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +Compute routing uses the namespace socket. A future cgroup routing integration requires a Toolkit interface that can bind the real container cgroup before startup, refresh the mapping on restart, and remove it on exit. Startup must fail if that mapping fails, and the memory hook must still run. [NVIDIA cgroup matching](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#cgroup-matching) + +## Daemon, recovery, and cleanup + +The manager takes an exclusive lock on the dedicated state directory. It records the state version, Pod and container identity, GPU set, limits, SM options, resource names, and preparation status in `allocations.json`. It uses the node's Pod list to reconcile these records with the daemon. + +On a drained daemon, the manager sets `compute-policy` and verifies the result. For `server-timeslice`, the plugin checks each GPU through NVML and makes only GPUs in `DEFAULT` compute mode eligible. The manager validates the actual GPU set when creating a workload server. Restoring the default yield requires the operator to drain the node and restart the host daemon. With live allocations it checks the existing policy and weights. Driver policy cannot be changed while a server exists, and a server weight cannot be changed while it has clients. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +Health and recovery use individual `get` fields for configuration and use r620 CSV inventories to establish resource presence and check static partition attributes. The 620.70.06 `sm-partition list --all` CSV includes a contract timeout field before status. An unknown format, query timeout, or disconnect stops the affected operation with its state preserved. Daemon or global policy failures mark all GPUs unhealthy. An allocation health check failure affects only the GPUs in that allocation. Read-only health checks run outside the allocation locks. + +Recovery reuses valid resources for live Pods. A live allocation from the old state format, an incompatible compute strategy, or a lost daemon with live allocations blocks startup. When the daemon survives, a manager restart can reconnect and validate the existing allocation. The host service owns the daemon lifecycle. If the host daemon exits, affected workloads require recovery. + +Reclamation deletes the namespace without force, deletes its partitions, deletes the server once it has no remaining allocations, then removes the allocation's CDI entry and state. r620 refuses namespace deletion while clients are attached. Failed cleanup retains its record for retry. Normal container restarts keep their allocation because kubelet can reuse its Allocate response. diff --git a/docs/userguide/nvidia-device/mps-v3.md b/docs/userguide/nvidia-device/mps-v3.md new file mode 100644 index 000000000..40b9f5750 --- /dev/null +++ b/docs/userguide/nvidia-device/mps-v3.md @@ -0,0 +1,162 @@ +--- +title: Share NVIDIA GPUs with MPS v3 +description: Configure MPS v3 in HAMi and request GPU memory, compute caps, static SM partitions, or weighted timeslices. +sidebar_label: MPS v3 +--- + +The `mps-v3` backend uses the NVIDIA Container Toolkit CDI hook to apply each container's GPU memory limit to its cgroup. CUDA processes connect to their assigned MPS namespace for compute sharing. This backend uses the existing HAMi GPU, memory, and core resources. + +The HAMi implementation is tracked in [HAMi PR #3443](https://github.com/Project-HAMi/HAMi/pull/3443). + +## NVIDIA device plugin compatibility + +HAMi enables r620 MPS v3 with `operatingmode: mps-v3`. It supports per-container GPU memory limits through CDI and offers namespace caps, static SM partitions with optional borrowing, and weighted server timeslices. + +The upstream NVIDIA device plugin's experimental `sharing.mps` uses Legacy MPS to expose equal-sized GPU replicas. That setting is unavailable in HAMi, where the node operating mode for MPS is `mps-v3`. [NVIDIA device plugin MPS documentation](https://github.com/NVIDIA/k8s-device-plugin/blob/v0.20.1/README.md#with-cuda-mps) + +## Prepare the node + +- Use Linux with cgroup v2, NVIDIA driver **r620**, and NVIDIA Container Toolkit **1.20.1 or later** with the `apply-cuda-memory-limits` hook enabled. Use application images with a CUDA runtime compatible with the installed driver. +- Disable MIG before enrolling the node. Configure a CDI-capable container runtime and UUID device IDs. Set device memory and core scaling to `1` on MPS v3 nodes. +- Make the driver-matched `nvidia-cuda-mps-control` and its libraries available inside the privileged device plugin, which uses the host PID namespace. Set `devicePlugin.mpsV3.binary` to the path visible inside the plugin. The chart's host `/usr/bin` mount is visible at `/usrbin`. With GPU Operator, locate the binary in the configured driver installation. +- Verify that the published CDI device for each GPU includes the memory hook and that the hook path is executable on the host. HAMi checks the published spec before registering devices. The Toolkit and driver manage the memory controller. [NVIDIA memory partitioning](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) + +```bash +stat -fc %T /sys/fs/cgroup +nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv +``` + +## Run the MPS daemon on the host + +Start a dedicated daemon on each MPS v3 node before enabling the device plugin. This example systemd unit uses the host binary path. The directories must match the Chart setting `devicePlugin.mpsV3.root`. + +```ini +[Unit] +Description=HAMi NVIDIA MPS v3 daemon + +[Service] +Type=simple +Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe +Environment=CUDA_MPS_LOG_DIRECTORY=/var/run/hami/mps-v3/log +ExecStartPre=/usr/bin/mkdir -p /var/run/hami/mps-v3/pipe /var/run/hami/mps-v3/log +ExecStart=/usr/bin/nvidia-cuda-mps-control -p 3 -f -m + +[Install] +WantedBy=multi-user.target +``` + +Save the unit as `/etc/systemd/system/hami-mps-v3.service` and run the following on the node. + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now hami-mps-v3.service +sudo env CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe \ + nvidia-cuda-mps-control -p 3 compute-policy get timeslice +``` + +The device plugin connects to the running daemon. Drain MPS workloads before stopping or restarting the host service. + +## Configure a node + +Select the node and choose its compute strategy in the Helm overrides. This example enables `server-timeslice` on one node. + +```yaml +devicePlugin: + hostPID: true + deviceListStrategy: cdi-cri + mpsV3: + enabled: true + binary: /usrbin/nvidia-cuda-mps-control + nodeConfiguration: + config: | + { + "nodeconfig": [ + { + "name": "gpu-node-01", + "operatingmode": "mps-v3", + "devicememoryscaling": 1, + "devicecorescaling": 1, + "mpsv3": { + "computeMode": "server-timeslice", + "timeslice": "long", + "timesliceYieldLevel": 1 + } + } + ] + } +``` + +To select the default `namespace-cap` strategy, omit the `mpsv3` object or use `{"computeMode":"namespace-cap"}`. Each selected `mpsv3` object replaces the previous one. Enable `mpsV3.enabled` to mount the node-local state and pipe directories. Choose `cdi-cri` when the kubelet and CRI support CDI devices, or `cdi-annotations` when the runtime consumes CDI annotations. Use only CDI strategies on these nodes. + +## Request compute and memory + +Set `nvidia.com/vgpu-mode: mps-v3` on the Pod. The optional `hami.io/mps-compute-mode` annotation selects the node's compute strategy and defaults to `namespace-cap`. Both strategies reserve core units from a 100-unit budget on each GPU. A resolved `gpucores` value of zero reserves 100 units. Use `nvidia.com/gpumem` or `nvidia.com/gpumem-percentage` according to the [GPU memory request rules](./specify-device-memory-usage.md). The Toolkit receives the final allocation in MiB as both the request and the hard limit. + +| Strategy | Meaning of `nvidia.com/gpucores: 25` | +| --- | --- | +| `namespace-cap` | The namespace uses an active-thread percentage of 25. CUDA may round the available compute resources to a hardware-supported amount. | +| `server-timeslice` | The container's server uses 25% of the daemon's base compute timeslice. Actual GPU time and throughput depend on which servers have work ready to run. | + +For example, this Pod uses 4 GiB of GPU memory and a timeslice of 25% of the configured base duration. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mps-training + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-compute-mode: server-timeslice +spec: + restartPolicy: Never + containers: + - name: training + image: your-cuda-application:tag + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 25 +``` + +Use `nvidia.com/gpu: 2` for two GPUs. Each GPU in a container must receive the same final memory and compute allocation. GPU sets can be identical or disjoint across containers. A request for GPUs A and B conflicts with an existing allocation for A alone. + +For `server-timeslice`, `mpsv3.timeslice` accepts `default` (the device's default), `short` (10 ms), `medium` (20 ms), or `long` (50 ms). The optional `mpsv3.timesliceYieldLevel` accepts `1` or `2`. The driver may reject percentages that produce a slice below the device's minimum. [NVIDIA weighted timeslicing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +## Request static SMs and optional borrowing + +For a `namespace-cap` Pod, add the following annotations and omit `gpucores` if the SM count is the only compute request. + +```yaml +metadata: + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-sm-count: "16" + hami.io/mps-sm-shareable: "true" + hami.io/mps-sm-max-chunks: "16" +``` + +- `hami.io/mps-sm-count` applies to each GPU of every GPU container in the Pod. An Ampere discrete GPU uses 2 SMs per chunk, so the example reserves eight chunks. Hopper and newer discrete GPUs use 8 SMs per chunk. Integrated GPUs use 2 SMs per chunk. A fixed partition can also use the device's remainder if it yields the exact requested count. +- `hami.io/mps-sm-shareable` allows the driver to lend the partition's idle SMs while it has no attached clients. Its default is `false`. +- `hami.io/mps-sm-max-chunks` caps the **total** chunks the partition may use, including its base reservation. The default `0` disables borrowing. A positive integer must be between the base and total chunks on every allocated GPU. Use `unlimited` to let the driver use available capacity without a configured cap. +- Borrowing or lending requires a base reservation made of whole chunks. The scheduler and quota account for the base reservation as `ceil(requested SMs / total SMs × 100)`. An explicit `gpucores` value must cover this charge. The driver controls how many additional SMs can be used at runtime. + +HAMi binds each partition to the namespace by GPU UUID. The namespace uses an active-thread percentage of 100 in this case. A physical GPU can serve either static SM partitions or namespace percentage allocations at a given time. The driver's ability to create another partition with clients already attached depends on the environment. HAMi returns the driver's error and rolls back the attempted allocation if creation fails. [NVIDIA partition load balancing](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#sm-partition-load-balancing), [NVIDIA namespace assignments](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +## Upgrade, switch strategy, or diagnose a failure + +1. Prevent new workloads from landing on the node and drain its MPS v3 workloads. For an upgrade from the earlier backend, let the old manager reclaim its resources before replacing it. +2. Apply the new driver, plugin, or node policy after reclamation. The same drain applies before changing `computeMode`, the base timeslice, the yield level, or the state root. +3. Confirm that the plugin reports healthy devices and run a workload with the intended strategy before reopening the node to GPU workloads. + +| Symptom | What to check | +| --- | --- | +| The node rejects MPS v3 during startup | Check driver r620, cgroup v2, non-MIG GPUs, UUID device IDs, scaling, the MPS control binary, and the published CDI hook. | +| A Pod cannot find a compatible GPU | Check the Pod and node strategies, available core and memory budget, the SM chunk geometry, and existing GPU sets. | +| Timeslice setup or readback fails | Check the target GPU and driver with the selected base timeslice. The driver can reject unsupported settings. Each GPU must use `DEFAULT` compute mode for `server-timeslice`. | +| Cleanup is waiting for clients | Let clients exit. HAMi removes the namespace first, then the partition and the last unused server. | +| The daemon is lost with live allocations | Investigate the daemon and restart affected workloads through Kubernetes. Preserve the allocation state while the Pods are live. | + +Each container receives a read-only mount of its namespace socket directory at `/run/hami-mps`. Compute settings apply to CUDA processes that attach through that socket. The MPS policies do not isolate memory bandwidth or provide MIG-style fault containment. Container restarts retain their allocations. Manager recovery requires the daemon and its resources to be available for live workloads. + +See the [MPS v3 backend design](../../developers/mps-v3-design.md) for allocation and recovery details. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md new file mode 100644 index 000000000..858450b41 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/mps-v3-design.md @@ -0,0 +1,99 @@ +--- +title: NVIDIA MPS v3 后端设计 +description: 介绍 HAMi MPS v3 后端的架构、GPU 调度、分配、恢复和回收。 +sidebar_label: MPS v3 设计 +--- + +HAMi 使用 NVIDIA Container Toolkit 的 `apply-cuda-memory-limits` CDI hook 限制 GPU 显存,并通过 MPS v3 共享算力。后端仅支持 r620 驱动分支,启动时检查专用 daemon 的命令接口。部署和工作负载示例见[用户指南](../userguide/nvidia-device/mps-v3.md)。 + +## 架构 + +```mermaid +flowchart TB + subgraph HAMi["HAMi 内部"] + direction LR + S["调度器"] --> D["设备插件"] + D --> M["MPS 管理器"] + M <--> ST["节点状态"] + D --> G["物理 GPU CDI"] + M --> A["分配 CDI"] + end + subgraph External["外部组件"] + direction LR + N["MPS daemon"] + R["容器运行时"] + R --> W["CUDA 应用"] + W -->|namespace socket| N + end + M -->|MPS control CLI| N + A --> R + G --> R + D -.->|CDI device ID| R + style HAMi fill:#eaf2ff,stroke:#4073ad,color:#172e4a + style External fill:#e8f5ee,stroke:#3f805d,color:#183c2a + classDef internal fill:#f7faff,stroke:#4073ad,color:#172e4a + classDef external fill:#f5fcf7,stroke:#3f805d,color:#183c2a + class S,D,M,ST,A internal + class N,G,R,W external +``` + +调度器匹配节点策略并核算每卡额度。设备插件读取分配注解,由 MPS 管理器验证资源归属,并在节点上记录分配意图和完成状态。HAMi 生成每次分配的专属 CDI,提供只读 socket 挂载、路由和显存环境变量。容器运行时应用其中的挂载和环境变量,CUDA 客户端通过挂载的 socket 连接 NVIDIA MPS namespace。设备插件调用 NVIDIA Container Toolkit 的库生成并发布物理 GPU CDI。该配置包含显存 hook,由容器运行时调用。 + +## 调度契约 + +设备插件上报 `mps-v3`、节点算力策略、SM 几何信息和已分配的 GPU 集合。调度器按 `hami.io/mps-compute-mode` 匹配节点策略,Pod 缺省为 `namespace-cap`。两种策略沿用每卡 100 单位的算力预算和现有配额。百分比或时间片请求的算力解析结果为 0 时,按 100 单位记账。 + +| 策略 | Server 身份 | 运行时配置 | +| --- | --- | --- | +| `namespace-cap` | 排序后的 GPU UUID 集合 | 每个 Pod UID 和容器名对应一个 namespace,设置活跃线程百分比或逐卡绑定静态分区。 | +| `server-timeslice` | Pod UID、容器名和排序后的 GPU UUID 集合 | 每个 GPU 容器使用独立 server。容器启动前写入并回读 `timeslice-pct`,namespace 百分比设为 100。 | + +同一容器在每张卡上使用相同的显存和算力额度。相同 GPU 集合在 `namespace-cap` 下可共用 server,在 `server-timeslice` 下可运行多个独立 server。调度器和管理器均拒绝部分重叠的集合。同一 GPU 不能混用静态 SM 分区和 namespace 百分比。 + +静态分区以基础 SM 数核算。创建静态分区前,管理器核对驱动报告的总 SM 数和 chunk 数。调度器按 `ceil(基础 SM 数 / 总 SM 数 × 100)` 记账。开启借出或借入时,基础额度须由整 chunk 组成,有限峰值须覆盖基础额度并符合每卡容量。驱动在运行时决定借用量,调度预留保持基础额度。 + +## 分配与容器启动 + +### 分配流程 + +```mermaid +flowchart TD + V["校验请求"] --> I["记录意图"] + I --> B{"算力策略"} + B -->|server-timeslice| T["创建加权 server"] + B -->|namespace-cap| C["获取共享 server"] + T --> TN["创建 namespace"] + C --> CN["配置 namespace"] + TN --> Q["回读校验"] + CN --> Q + Q --> P["发布 CDI 并落盘"] + P --> A["返回 Allocate 响应"] + T -.->|操作失败| X["回滚"] + C -.-> X + TN -.-> X + CN -.-> X + Q -.-> X + P -.-> X +``` + +1. 校验 GPU 集合、策略、显存和算力设置。复用 server 前验证其归属。 +2. 在创建资源前将稳定的资源名称和分配意图写入节点状态。 +3. 创建或复用 server。新时间片 server 先设置权重,再创建 namespace。静态 SM 请求先按卡创建分区,再逐卡单独执行 `namespace set --device= --sm-partition=`。 +4. 回读 GPU 集合、namespace 百分比、时间片权重和 SM 绑定,检查 socket。静态分区校验覆盖基础 chunks、余数、借出和借入选项。借用造成的有效 SM 数变化属于正常运行行为。 +5. 发布专属 CDI 并持久化准备完成状态,再返回分配结果。请求内任一容器分配失败时,回滚本次新建的资源。 + +分配专属 CDI 提供只读 namespace socket 挂载、`CUDA_MPS_PIPE_DIRECTORY` 和整数 MiB 的 `NVIDIA_GPU_MEMORY_REQUEST`、`NVIDIA_GPU_MEMORY_LIMIT`。物理 GPU CDI 的 `createRuntime` hook 在应用启动前配置显存上限。服务端的静态分区绑定优先于客户端设置的分区标识。[NVIDIA namespace 绑定](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +本版通过专属 namespace socket 路由算力。接入 cgroup 路由需要 Toolkit 提供正式接口,在容器启动前绑定真实 cgroup、重启时更新映射、退出时清理。绑定失败须阻止启动,并保留显存 hook 的正常执行。[NVIDIA cgroup 路由](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#cgroup-matching) + +## Daemon、恢复与回收 + +管理器对专用状态目录加独占锁,在 `allocations.json` 记录版本、Pod 与容器身份、GPU 集合、额度、SM 选项、资源名称和准备状态。启动时根据权威节点 Pod 列表和 daemon 状态恢复分配。 + +排空后,管理器设置并核对 `compute-policy`。`server-timeslice` 节点通过 NVML 逐卡检查 compute mode,仅允许使用 `DEFAULT` 模式的 GPU。实际 GPU 集合在创建工作负载的 server 时验证。恢复默认 yield 时,运维需先排空节点并重启宿主机 daemon。存在活跃分配时,管理器只核对既有策略和权重。daemon 级策略需要在没有 server 时变更,server 权重需要在没有客户端时变更。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +健康检查和恢复优先使用 `get` 字段回读配置。资源是否存在、静态分区的具体设置通过 r620 CSV inventory 核对。620.70.06 的 `sm-partition list --all` 在状态列前还有一列 contract timeout。未知格式、查询超时或断连会停止当前操作并保留记录。daemon 或全局策略异常影响整节点,单个分配的健康异常仅影响其 GPU。只读健康检查在分配锁外执行。 + +恢复复用仍与记录一致的资源。遇到仍存活的旧版分配、策略不匹配或 daemon 丢失且仍有活跃分配时,管理器拒绝接管。仅重启管理器时,可以重新连接仍存活的 daemon。daemon 由宿主机服务托管,插件重启后重新连接。如果宿主机 daemon 退出,则需要恢复受影响的工作负载。 + +回收依次执行非强制删除 namespace、删除分区、删除无引用 server,再移除 CDI 和状态。r620 会拒绝删除仍有客户端的 namespace,失败记录保留供重试。普通容器重启保留原分配,以便 kubelet 复用 Allocate 响应。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md new file mode 100644 index 000000000..6c237427b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/mps-v3.md @@ -0,0 +1,162 @@ +--- +title: 使用 MPS v3 共享 NVIDIA GPU +description: 介绍如何在 HAMi 中配置 MPS v3,并申请显存、算力上限、静态 SM 分区和加权时间片。 +sidebar_label: MPS v3 +--- + +`mps-v3` 后端通过 NVIDIA Container Toolkit 的 CDI hook 为每个容器的 cgroup 设置显存上限。CUDA 进程连接分配给容器的 MPS namespace,按节点配置的策略共享算力。GPU 数量、显存和算力继续使用现有的 HAMi 资源名。 + +HAMi 实现见 [HAMi PR #3443](https://github.com/Project-HAMi/HAMi/pull/3443)。 + +## 与 NVIDIA device plugin 的关系 + +HAMi 通过 `operatingmode: mps-v3` 启用 r620 MPS v3,用 CDI 设置容器显存上限,并提供 namespace 算力上限、可选借用的静态 SM 分区和 server 加权时间片。 + +NVIDIA 上游 device plugin 的实验性 `sharing.mps` 使用旧版 MPS,将 GPU 暴露为等额副本。HAMi 不支持该配置,启用 MPS 时使用节点模式 `mps-v3`。[NVIDIA device plugin MPS 文档](https://github.com/NVIDIA/k8s-device-plugin/blob/v0.20.1/README.md#with-cuda-mps) + +## 准备节点 + +- 使用 cgroup v2、**NVIDIA r620 驱动**和 **NVIDIA Container Toolkit 1.20.1 或更新版本**。启用 `apply-cuda-memory-limits` hook。应用镜像应使用与节点驱动兼容的 CUDA 运行时。 +- 启用后端前先关闭 MIG。使用支持 CDI 的容器运行时,以 UUID 标识 GPU,并将节点的显存和算力缩放系数设为 `1`。 +- 确认特权 device plugin 使用主机 PID 命名空间,并能访问与驱动配套的 `nvidia-cuda-mps-control` 和驱动库。Chart 将主机的 `/usr/bin` 挂到插件容器内的 `/usrbin`。使用 GPU Operator 时,请按驱动的实际安装位置配置二进制路径。 +- 检查已经发布的物理 GPU CDI 设备包含显存 hook,且 hook 路径在主机上可执行。HAMi 在注册设备前会检查该配置。显存控制器由 Toolkit 和驱动管理。[NVIDIA 显存分区文档](https://docs.nvidia.com/deploy/mps/mpsv3-memory-partitioning.html) + +```bash +stat -fc %T /sys/fs/cgroup +nvidia-smi --query-gpu=uuid,driver_version,mig.mode.current,mig.mode.pending --format=csv +``` + +## 在宿主机运行 MPS daemon + +在启用 MPS v3 的节点上先启动专用 daemon,再启用设备插件。下面是 systemd unit 示例。二进制使用宿主机路径,目录须与 Chart 的 `devicePlugin.mpsV3.root` 一致。 + +```ini +[Unit] +Description=HAMi NVIDIA MPS v3 daemon + +[Service] +Type=simple +Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe +Environment=CUDA_MPS_LOG_DIRECTORY=/var/run/hami/mps-v3/log +ExecStartPre=/usr/bin/mkdir -p /var/run/hami/mps-v3/pipe /var/run/hami/mps-v3/log +ExecStart=/usr/bin/nvidia-cuda-mps-control -p 3 -f -m + +[Install] +WantedBy=multi-user.target +``` + +保存为 `/etc/systemd/system/hami-mps-v3.service` 后,在节点上执行: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now hami-mps-v3.service +sudo env CUDA_MPS_PIPE_DIRECTORY=/var/run/hami/mps-v3/pipe \ + nvidia-cuda-mps-control -p 3 compute-policy get timeslice +``` + +设备插件只连接已有 daemon。停止或重启节点服务前,先排空 MPS 工作负载。 + +## 配置节点 + +以下 Helm 配置在 `gpu-node-01` 上启用 `server-timeslice`。按实际节点名称和驱动路径修改。 + +```yaml +devicePlugin: + hostPID: true + deviceListStrategy: cdi-cri + mpsV3: + enabled: true + binary: /usrbin/nvidia-cuda-mps-control + nodeConfiguration: + config: | + { + "nodeconfig": [ + { + "name": "gpu-node-01", + "operatingmode": "mps-v3", + "devicememoryscaling": 1, + "devicecorescaling": 1, + "mpsv3": { + "computeMode": "server-timeslice", + "timeslice": "long", + "timesliceYieldLevel": 1 + } + } + ] + } +``` + +使用默认的 `namespace-cap` 时,省略 `mpsv3` 对象或写为 `{"computeMode":"namespace-cap"}`。多条匹配的节点配置按顺序应用,后面的 `mpsv3` 对象整体替换前一个。启用 `mpsV3.enabled` 后,Chart 会挂载节点状态和 socket 目录。按 kubelet 和运行时的 CDI 接入方式选择 `cdi-cri` 或 `cdi-annotations`。MPS v3 节点只接受 CDI 设备列表策略。 + +## 申请算力和显存 + +在 Pod 上设置 `nvidia.com/vgpu-mode: mps-v3`。用 `hami.io/mps-compute-mode` 选择节点的算力策略,缺省值为 `namespace-cap`。两种策略均按每卡 100 单位核算配额,解析得到的 `gpucores` 为 0 时按 100 记账。按[显存申请规则](./specify-device-memory-usage.md)使用 `nvidia.com/gpumem` 或 `nvidia.com/gpumem-percentage`。Toolkit 将最终分配的 MiB 数同时用于显存请求和硬上限。 + +| 策略 | `nvidia.com/gpucores: 25` 的含义 | +| --- | --- | +| `namespace-cap` | namespace 的活跃线程百分比上限为 25,硬件可以对可用算力取整。 | +| `server-timeslice` | 容器专属 server 的时间片长度为 daemon 基础时间片的 25%。实际 GPU 时间和吞吐取决于各 server 有多少任务就绪。 | + +以下 Pod 申请 4 GiB 显存和基础时间片长度的 25%。 + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mps-training + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-compute-mode: server-timeslice +spec: + restartPolicy: Never + containers: + - name: training + image: your-cuda-application:tag + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 25 +``` + +申请两张卡时使用 `nvidia.com/gpu: 2`。同一容器在每张卡上的最终显存和算力额度必须相同。不同容器可以使用完全相同或互不相交的 GPU 集合。若 GPU A 已被单卡分配使用,A+B 的请求会与现有分配冲突。 + +`server-timeslice` 的 `mpsv3.timeslice` 支持 `default`(设备默认值)、`short`(10 ms)、`medium`(20 ms)和 `long`(50 ms)。可选的 `mpsv3.timesliceYieldLevel` 接受 `1` 或 `2`。当计算后的时间片低于设备支持的最小值时,驱动会拒绝设置。[NVIDIA 加权时间片接口](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#weighted-timeslicing) + +## 申请静态 SM 和可选借用 + +在 `namespace-cap` Pod 上添加以下注解。仅通过 SM 数申请算力时,可省略 `gpucores`。 + +```yaml +metadata: + annotations: + nvidia.com/vgpu-mode: mps-v3 + hami.io/mps-sm-count: "16" + hami.io/mps-sm-shareable: "true" + hami.io/mps-sm-max-chunks: "16" +``` + +- `hami.io/mps-sm-count` 适用于 Pod 内每个 GPU 容器申请的每张卡。Ampere 独立 GPU 每个 chunk 为 2 SM,因此示例预留 8 个 chunk。Hopper 及更新的独立 GPU 每个 chunk 为 8 SM,集成 GPU 为 2 SM。固定分区也可以使用该设备的余数 SM,前提是能精确组成请求值。 +- `hami.io/mps-sm-shareable` 允许驱动在分区没有客户端时借出空闲 SM。默认 `false`。 +- `hami.io/mps-sm-max-chunks` 是分区能使用的**总** chunk 数上限,包含基础额度。默认 `0` 表示禁止借入。正整数上限必须在每卡的基础 chunk 数与该卡总 chunk 数之间。`unlimited` 允许驱动使用可借容量。 +- 启用借出或借入时,基础额度必须是整 chunk。调度和配额按 `ceil(基础 SM 数 / GPU 总 SM 数 × 100)` 记账。显式设置的 `gpucores` 必须覆盖该值。运行时实际借用量由驱动决定。 + +HAMi 按 GPU UUID 将分区绑定到 namespace,并将 namespace 的活跃线程百分比设为 100。同一 GPU 同时只支持静态 SM 分区或 namespace 百分比。GPU 有活跃客户端时能否创建新分区取决于驱动与设备。若驱动拒绝,HAMi 会报告错误并回滚该次分配。[NVIDIA SM 借用](https://docs.nvidia.com/deploy/mps/620/mpsv3-interface.html#sm-partition-load-balancing)、[NVIDIA namespace 绑定](https://docs.nvidia.com/deploy/mps/620/mpsv3-namespaces.html#sm-partition-assignments) + +## 升级、切换策略和排查 + +1. 阻止新任务进入节点,并排空其 MPS v3 工作负载。从旧版升级时,先由旧管理器回收已有分配。 +2. 确认资源已回收,再变更驱动、插件或节点策略。修改 `computeMode`、基础时间片、yield 或状态目录也需要排空。 +3. 确认插件上报健康,运行符合新策略的验证任务,再恢复节点调度。 + +| 现象 | 检查项 | +| --- | --- | +| MPS v3 节点启动失败 | 检查 r620 驱动、cgroup v2、非 MIG GPU、UUID 设备 ID、缩放系数、MPS 工具和已发布的 CDI hook。 | +| Pod 无法匹配 GPU | 检查 Pod 与节点策略、算力和显存余量、SM chunk 和现有 GPU 集合。 | +| 时间片设置或回读失败 | 检查目标 GPU、驱动和基础时间片。`server-timeslice` 要求 GPU 使用 `DEFAULT` compute mode。 | +| 资源回收等待客户端 | 等待 CUDA 客户端退出。HAMi 依次删除 namespace、分区和最后一个无引用的 server。 | +| daemon 丢失且仍有存活分配 | 排查 daemon 后通过 Kubernetes 重建受影响的工作负载。在 Pod 存活期间保留分配状态。 | + +容器仅在 `/run/hami-mps` 看到自己的 namespace socket 目录。算力设置对通过该 socket 接入的 CUDA 进程生效。MPS 不提供显存带宽或 MIG 级别的故障隔离。普通容器重启保留原分配。若仍有存活工作负载,管理器恢复依赖原 daemon 及其资源。 + +分配流程和恢复条件见 [MPS v3 后端设计](../../developers/mps-v3-design.md)。 diff --git a/sidebars.js b/sidebars.js index 0ca80af62..fded8b3ee 100644 --- a/sidebars.js +++ b/sidebars.js @@ -31,6 +31,7 @@ const vendors = [ "scheduling-policy", "numa-alignment", "dynamic-mig-support", + "mps-v3", "dynamic-resource-allocation", "using-resourcequota", ]), @@ -357,6 +358,7 @@ module.exports = { "developers/dynamic-mig", "developers/dynamic-mig-lifecycle", "developers/dynamic-mig-migration", + "developers/mps-v3-design", "developers/gpu-topology-scheduling", "developers/kunlunxin-topology", "developers/profiling-scheduler",