User problem
TauGrid operators and researchers need detailed DCGM telemetry for GPU troubleshooting and experiment correlation, including per-link NVLink counters, raw NVLink CRC counters, and power/thermal throttling residency. These fields are valuable, but they are not uniformly available through the AKS-managed DCGM exporter and do not independently indicate unhealthy hardware. Treating them as universal required health fields can create unsupported configuration, excessive metric cardinality, or false unhealthy classifications.
Desired outcome
Establish and document a deliberate GPU metric model that clearly separates:
- Baseline health fields reliably available from the AKS-managed exporter.
- Extended health fields requiring an explicitly configured exporter and suitable health semantics.
- Deep diagnostic and performance telemetry used for dashboards, experiment correlation, and troubleshooting without automatically affecting GPU health.
The resulting contract should define field availability, metric types, cardinality, retention, coverage behavior, GPU-generation applicability, and evidence-based alert thresholds.
Proposed approach
Review the GPU monitoring chart, collector rules, dashboards, and exporter integration end to end. Specifically:
- Inventory AKS-managed, NVIDIA default, and TauGrid-requested DCGM fields.
- Evaluate per-link NVLink CRC/data/replay metrics across supported GPU generations and topologies.
- Determine how power and thermal violation-duration counters should be visualized and correlated with workloads.
- Separate telemetry collection from health classification and fail-closed metric coverage.
- Consider optional collector profiles such as
baseline, extended-health, and deep-diagnostics rather than one universal CSV.
- Define model-specific rate or threshold policies before promoting diagnostic counters into alerts.
- Document how a separately managed exporter can consume optional TauGrid CSV profiles without modifying the AKS-managed exporter.
- Add focused validation for rendered profiles, metric types, and rule/profile compatibility.
Any implementation should preserve default AKS compatibility and avoid deploying overlapping GPU drivers, device plugins, or host engines.
Alternatives considered
- Include every available DCGM field in the default CSV. This risks unsupported fields, high cardinality, and inconsistent behavior across GPU generations.
- Remove the diagnostic fields entirely. This loses useful evidence for troubleshooting and performance analysis.
- Treat every counter increment as a health failure. This is not appropriate for CRC rates or normal power/thermal throttling behavior without hardware- and workload-specific thresholds.
Confirmation
User problem
TauGrid operators and researchers need detailed DCGM telemetry for GPU troubleshooting and experiment correlation, including per-link NVLink counters, raw NVLink CRC counters, and power/thermal throttling residency. These fields are valuable, but they are not uniformly available through the AKS-managed DCGM exporter and do not independently indicate unhealthy hardware. Treating them as universal required health fields can create unsupported configuration, excessive metric cardinality, or false unhealthy classifications.
Desired outcome
Establish and document a deliberate GPU metric model that clearly separates:
The resulting contract should define field availability, metric types, cardinality, retention, coverage behavior, GPU-generation applicability, and evidence-based alert thresholds.
Proposed approach
Review the GPU monitoring chart, collector rules, dashboards, and exporter integration end to end. Specifically:
baseline,extended-health, anddeep-diagnosticsrather than one universal CSV.Any implementation should preserve default AKS compatibility and avoid deploying overlapping GPU drivers, device plugins, or host engines.
Alternatives considered
Confirmation