[P2P] Bind one-sided IPC test tensors to the endpoint device - #1050
Conversation
test_engine_onesided_ipc.py built its endpoint on local_gpu_idx=rank but allocated every payload and reference tensor with an explicit device="cuda:0" after torch.cuda.set_device(0). With two processes sharing one visible device list, rank 1 therefore had its endpoint on GPU 1 while all of its payload tensors were on GPU 0, so no transfer in the suite exercised a cross-GPU payload path and the run stayed green. Resolve the device once per process, use it for the endpoint, for every GPU tensor and for the reference tensors, and assert that the endpoint BDF matches the payload device BDF plus that the two ranks resolve to different physical GPUs. Verified on 4x L20 (PCIe, no NVLink): pairs 0/1, 1/0 and 0/3 (cross NUMA island) each run 24/24 PASSED with the endpoint and payload on 34:00.0/35:00.0 respectively, and the pre-fix binding is rejected by the new check.
| payload_bdf = _visible_bdf_by_uuid().get(uuid) | ||
| assert payload_bdf is not None, f"no nvidia-smi BDF for GPU uuid {uuid}" | ||
|
|
||
| assert payload_bdf == endpoint_bdf, ( |
There was a problem hiding this comment.
[P2] Normalize PCI-address case before comparing
The engine stores GPU BDFs in lowercase through normalize_pci_bus_id, while nvidia-smi emits uppercase hexadecimal digits. _normalize_bdf only removes the domain, so a correctly bound GPU such as endpoint 0000:ca:00.0 / nvidia-smi 00000000:CA:00.0 fails this assertion before any IPC test runs. I reproduced this using the new helpers with mocked metadata. The numeric-only 34/35/37 addresses in the reported validation do not exercise it. Please canonicalize case on both sides.
|
|
||
| def _normalize_bdf(bdf: str) -> str: | ||
| """Drop the PCI domain so UCCL metadata and nvidia-smi are comparable.""" | ||
| return bdf.split(":", 1)[1] if bdf.count(":") >= 2 else bdf |
There was a problem hiding this comment.
[P2] Preserve the PCI domain in physical GPU identities
Dropping the domain makes distinct GPUs such as 0008:01:00.0 and 0009:01:00.0 both resolve to 01:00.0. These addresses occur on Grace Blackwell systems, so the final gathered[0] != gathered[1] assertion rejects a valid cross-GPU run. It also loses the ability to detect a binding mismatch that differs only in the domain. I reproduced the collision with the new helpers. Please normalize the domain's zero padding rather than dropping it, or use full GPU UUIDs for the cross-rank identity check.
| def _visible_bdf_by_uuid() -> dict: | ||
| """Map every visible GPU UUID to its PCI bus id (domain stripped).""" | ||
| out = subprocess.run( | ||
| ["nvidia-smi", "--query-gpu=uuid,pci.bus_id", "--format=csv,noheader"], |
There was a problem hiding this comment.
[P2] Keep the binding check usable on HIP systems
This now unconditionally runs nvidia-smi during startup. The P2P engine and this torch.cuda-based test also support AMD/HIP, where nvidia-smi is normally absent, so the new preflight raises FileNotFoundError before exercising any IPC operation. I reproduced that failure by making the subprocess call report a missing executable. Please retrieve the PCI address through the active GPU runtime or add an appropriate HIP path instead of imposing an NVIDIA-only executable dependency.
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
|
Fixed all three identity issues: canonical lowercase full-domain BDFs, preserved domain distinctions, and active CUDA/HIP runtime queries without
NCCL endpoint backend on this non-RDMA host; transfers use real CUDA IPC. HIP selection was mocked, not tested on AMD hardware. |
2 participants
Description
Bind every payload and reference tensor to the endpoint's process-local GPU ordinal. The test previously put rank 1's endpoint on GPU 1 while allocating its tensors on GPU 0, so it could report success without exercising the intended cross-GPU payload path.
The preflight compares canonical full PCI addresses: hexadecimal case and domain padding are normalized, while distinct domains remain distinct. It queries the active CUDA/HIP runtime directly and has no
nvidia-smidependency.Test result
2× L40S, torch 2.13.0+cu129, native extension built with CUDA 12.8 from this PR's engine sources. This host has no RDMA devices, so the endpoint uses
UCCL_P2P_TRANSPORT=nccl; all one-sided operations below use the actual CUDA IPC path.UCCL_P2P_TRANSPORT=nccl CUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=1 \ python -m torch.distributed.run --standalone --nproc-per-node=2 \ p2p/tests/test_engine_onesided_ipc.py # Repeated with CUDA_VISIBLE_DEVICES=1,0. python -m pytest -q p2p/tests/test_engine_device_binding.py0000:21:00.0and0000:81:00.0, correctly reversed with visibility orderHIP runtime selection is covered with mocks; AMD hardware was not available. These are correctness tests, with no throughput speedup claim. No engine or transport code changes are included.