[Feature]: Model interconnect requirements with topology-aware scheduling

[chokevin]

User problem

TauGrid currently turns RDMA.Enabled into a Kubernetes extended-resource request:

resources:
  requests:
    rdma/rdma_shared_device_a: "1"
  limits:
    rdma/rdma_shared_device_a: "1"

Kueue then requires matching ClusterQueue and ResourceFlavor quota. Workloads can remain inadmissible even when suitable InfiniBand-connected nodes exist.

RDMA should default to a placement requirement, not consumable queue quota. Users should request an interconnect capability without knowing a device-plugin resource name. The topology must remain cluster-defined: for example, an NVL72 rack may be an NVLink/NVSwitch domain inside a broader IB/RDMA fabric domain.

Desired outcome

Let users or resource profiles request an interconnect capability. TauGrid should place all workload participants within the narrowest compatible Kueue topology domain:

• NVLink/NVSwitch workloads stay within one compatible local domain.
• RDMA workloads stay within one IB/RDMA fabric domain and may span racks.
• Workloads without an interconnect requirement use normal queue placement.

By default, RDMA must not add or require rdma/rdma_shared_device_a in pod resources, ResourceFlavors, or ClusterQueue quota.

Proposed approach

Represent interconnect capabilities in cluster topology, for example:

IB/RDMA fabric
└── rack or NVLink domain
    └── node

When a workload requests a capability, TauGrid should:

1. Select the narrowest configured topology domain that provides it.
2. Apply consistent topology-aware constraints to every participant.
3. Keep the workload queued with a clear reason if no compatible domain has enough capacity.
4. Support explicit device-resource requests only as an advanced cluster-specific override.

Keep runtime setup such as security context, NCCL configuration, and shared memory separate from topology placement and device allocation.

Current coupling

addRDMAResources injects a device-plugin-specific resource whenever RDMA is enabled. This couples runtime setup, device allocation, queue quota, and topology placement behind one switch.

Related but distinct: #290 added missing Job-side RDMA runtime/security-context support.

Acceptance criteria

• Users can request RDMA or another supported interconnect without specifying an extended-resource name.
• Default rendered pods and ClusterQueues do not require rdma/rdma_shared_device_a.
• Cluster topology can distinguish node, NVLink/rack, and IB/RDMA fabric scopes.
• TauGrid selects the narrowest compatible domain and applies it consistently to all participants.
• Missing or insufficient compatible capacity produces a useful queue or validation reason.
• Non-RDMA workloads remain unconstrained by RDMA topology.
• RDMA runtime setup still works after extended-resource injection is removed.
• Explicit device-resource overrides remain supported.
• Tests cover single-node, intra-NVLink-domain, multi-rack RDMA, insufficient capacity, explicit override, and non-RDMA cases.

Confirmation

• I searched existing issues and roadmap items for this request.
• This request does not include confidential or security-sensitive information.

[chokevin]

Basically the goal should be that the taugrid controller can autodetect rdma or NVL nodes and make sure they are under the right topology such that when a user creates a job expecting a certain topology they can request it under TAS