observabilitygpuperformance

Observability Patterns for GPU-Accelerated Test Fleets with NVLink

mmytest

2026-02-19

10 min read

Correlate GPU, NVLink, and storage telemetry across ephemeral clusters to find regressions fast and lower test costs.

Hook — Stop chasing ghosts across ephemeral GPU clusters

If your CI runs slow, tests are flaky, and performance regressions only become obvious after a costly rollout, you don’t have an observability problem — you have a correlation problem. In 2026, teams run ephemeral GPU-accelerated test fleets (multi-GPU nodes connected with NVLink) for short-lived model runs, inference tests, and hardware-sensitive performance suites. Collecting raw GPU metrics alone is no longer enough. You must collect, correlate, and analyze GPU metrics, NVLink traffic, and storage I/O as a single telemetry fabric to find regressions quickly and cheaply.

The single most important idea

Correlate by context, not by clock. Attach the same minimal context (CI job ID, test run ID, pod/container ID, GPU UUID, NVLink link ID, storage device name) to every metric, trace, and log. With consistent context and synchronized timestamps you can pivot from a slow test to the exact GPU, the saturated NVLink link, or the slow storage device that caused the regression — in minutes instead of days.

What’s changed in 2025–2026 and why this matters

NVLink adoption expanded beyond traditional x86 servers — announcements such as SiFive integrating NVLink Fusion with RISC-V IP (Jan 2026) show interconnects are being embedded into new CPU architectures. These changes increase NVLink topology variance inside test fleets and raise the need for link-level telemetry.
GPU tooling improved: NVIDIA’s DCGM, NVML, Nsight tooling, and per-GPU counters are richer (late 2025), and vendors expose NVLink-specific metrics more consistently. But this exposes a new challenge: more metrics at higher frequency = higher collection costs.
Ephemeral clusters and autoscaling patterns (Karpenter, cluster autoscaler) are standard in CI/CD, so telemetry must be robust to frequent node churn and preemption.

Telemetry you need to collect (and why)

At minimum, collect these correlated streams from each ephemeral node:

GPU metrics — utilization, memory usage, temperature, power draw, ECC errors, CUDA context counts. Source: NVIDIA DCGM / NVML.
NVLink traffic — per-link throughput (RX/TX), link utilization, link errors. NVLink saturation is a common hidden bottleneck for multi-GPU workloads.
Storage I/O — per-device throughput, IOPS, latency percentiles, queue depth. Many regressions are I/O-bound, not compute-bound.
Host metrics — CPU, interrupts, network, NUMA node balances, memory pressure.
Application traces — high-cardinality trace IDs propagated from CI steps into the test process, covering key function calls and major I/O events.
Logs — test harness logs, system dmesg (ECC/NVLink errors), kernel messages (I/O errors, driver warnings).

Architecture pattern: telemetry pipeline for ephemeral GPU fleets

Design a pipeline that is resilient to node churn, supports high-frequency metrics briefly, and is cost-aware for long-term retention.

Lightweight node collectors: run DCGM exporter (or PyNVML agent) + an eBPF-based I/O tracer on each node, packaged as a DaemonSet in Kubernetes or a systemd service for bare metal. Keep each agent stateless.
OpenTelemetry Collector: use an OTEL collector on each node or a sidecar to unify metrics, traces, and logs. Enrich telemetry with context (CI_JOB_ID, POD_NAME, GPU_UUID) before forwarding.
Short-term hot store: use a horizontally scalable metrics store (Cortex/VictoriaMetrics/Prometheus remote write cluster) plus a tracing backend (Grafana Tempo/Jaeger) for immediate investigation.
Long-term tiered storage: ship compressed rollups and traces to object storage (S3/MinIO) via Thanos/Tempo or a cold store for cost-effective retention.
Visualization & Alerting: Grafana dashboards combining metrics, NVLink graphs, and traces. Alerting for NVLink saturation, GPU ECC errors, and percentiles of storage latency that are deviation from baseline.

Why OpenTelemetry + DCGM is the winning combination

DCGM/NVML is the most reliable source of GPU and NVLink counters. OpenTelemetry provides a vendor-neutral way to attach context and forward metrics, traces, and logs together. That pairing lets you answer questions like:

"Which CI job produced the spike in NVLink TX at 12:03 UTC, and which GPU (UUID) and storage device were active then?"

Practical implementation: collectors, labeling, and sync

1) Node collectors and exporters

Deploy the following on each node:

DCGM exporter (docker image or DaemonSet). Configure to collect GPU and NVLink metrics at 1–5s resolution during tests, falling back to 30–60s at idle.
eBPF-based I/O monitor (bcc/Libbpf-based) to capture per-process block I/O latency and op counts. This lets you map storage I/O to the test process PID and then to the CI job context.
Node-level OTEL collector that receives metrics/traces/logs, enriches with environment variables (CI_JOB_ID, GIT_SHA), and remote-writes.

2) Context propagation and labels

Attach the following minimal context to every telemetry point:

ci.job.id — unique per pipeline execution
test.run.id — specific test scenario id
pod/container — Kubernetes metadata
gpu.uuid — GPU hardware identifier
nvlink.id — NVLink link identifier (if available)
storage.device — device name or block ID

Propagate these via environment variables or a sidecar injector. For containerized runs, CI runners should inject ci.job.id and test.run.id into the pod spec so collectors pick them up automatically.

3) Time synchronization

Synchronized timestamps are critical. Use NTP/PTP with a target max skew of 1-10ms across the fleet. If your cloud provider or on-prem hardware supports PTP, enable it. Otherwise, at minimum use chronyd with frequent polling. For eBPF and kernel-level events, use monotonic time mapping to wall clock in the OTEL collector to avoid cross-source timestamp drift.

Actionable config snippets

Below are minimal examples you can adapt. These are starting points — tweak sampling and retention for your fleet size and budget.

DCGM exporter (snippet)

# Run DCGM exporter; example command for a DaemonSet container
nvidia-dcgm-exporter --collectors=dcgm,pergpu --scrape-interval=5s

OTEL collector pipeline (high-level)

receivers:
  prometheus:
    config: 
exporters:
  otlp:
    endpoint: "otel-ingest:4317"
processors:
  batch:
  attributes:
    actions:
      - key: ci.job.id
        action: insert
        value: "${CI_JOB_ID}"
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [attributes, batch]
      exporters: [otlp]

PromQL patterns to correlate GPU, NVLink, and I/O

Use these example queries in Grafana panels. Replace label keys with your own naming conventions.

# GPU utilization (per pod)
avg by (pod, ci_job_id) (rate(dcgm_gpu_utilization_percent[30s]))

# NVLink throughput (per link)
sum by (nvlink_id, ci_job_id) (rate(dcgm_nvlink_throughput_bytes_total[30s]))

# Disk read latency (p99) for devices involved in a job
histogram_quantile(0.99, sum by (le, ci_job_id, storage_device) (rate(node_disk_read_seconds_bucket[5m])))

# Correlate: high NVLink throughput with falling GPU utilization
# Find jobs where NVLink throughput > X and GPU util < Y
(
  sum by (ci_job_id) (rate(dcgm_nvlink_throughput_bytes_total[30s])) > 1e9
) and
(
  avg by (ci_job_id) (rate(dcgm_gpu_utilization_percent[30s])) < 30
)

Alerting and SLOs for test fleets

Define SLOs that match your testing goals. Examples:

NVLink saturation SLO: fewer than 5% of test runs should experience sustained NVLink utilization > 90% for more than 30s.
GPU starvation SLO: CPU or storage-induced stalls should not reduce mean GPU utilization by > 50% of baseline for more than 10% of runs.
Storage latency SLO: p99 read latency below 20ms for test workloads.

Create alert policies with context-enriched annotations so alerts directly link to the CI job and test run ID for rapid investigation.

Cost optimization patterns

Telemetry is expensive at scale. Use these patterns to keep costs under control while still enabling fast regression detection.

Adaptive sampling — collect high-frequency metrics (1–5s) only while the CI job is running. Revert to low-frequency sampling (30–60s) afterwards.
Tiered storage — keep full-resolution metrics and traces in the hot store for 7–14 days, then downsample and store rollups in object storage.
Cardinality controls — avoid unbounded label cardinality. Limit labels like ci.branch to whitelists, and use hashed mappings for high-cardinality values.
Pre-filter heavy traces — capture full traces for failed or anomalous runs only; sample successful runs at 0.1–1%.
Use spot/preemptible GPUs for non-critical runs — but ensure telemetry agents gracefully handle preemption and forward buffered metrics before shutdown.

Advanced strategies: ML-assisted correlation and continuous profiling

Once you have cleanly correlated telemetry, you can add advanced layers:

Anomaly detection — run streaming anomaly detection on multi-dimensional features (GPU util, NVLink throughput, I/O latency) to surface regressions early. Use lightweight models in the ingest path to flag runs for full trace capture.
Continuous profiling — integrate continuous profilers (e.g., async-profiler, perf events, Nvidia Nsight sampling) that attach to runs marked anomalous. Store profiles alongside the CI job ID and make them linkable from alerts.
Automated root-cause hints — implement rule-based or ML models to generate candidate root causes (NVLink saturation, storage queueing, CPU stealing) and rank them by likelihood to speed human investigation.

Troubleshooting runbook (first 15 minutes)

When a test run fails or slows, query by ci.job.id across metrics, traces, logs. Look for NVLink spikes and storage p99 before GPU util drops.
If NVLink is saturated, identify the GPUs and the topology. Check for background replication or other tenants sharing the interconnect.
If storage latency spikes coincide, correlate process-level I/O from eBPF traces to the container PID and evaluate whether the run is I/O bound.
Capture a continuous profiler snapshot for 30s on the affected pod. Compare against baseline profile for the same test.run.id on a good run.
Open a ticket with annotated traces and profiles linking to the offending commit (GIT_SHA) for developer follow-up.

Real-world example (anonymized, pattern only)

An AI infra team running ephemeral multi-GPU tests noticed occasional long tail test failures that escaped unit tests. By instrumenting DCGM + eBPF + OTEL and enriching every telemetry point with ci.job.id they were able to pivot directly from a failing test to a single NVLink link with sustained 95% utilization and a p99 read latency spike on the attached NVMe pool. The fix was a scheduling change that avoided colocating two heavy NVLink traffic flows on the same host, cutting median regression triage time from hours to under 30 minutes. The change also reduced redundant full-resolution tracing by 80% via adaptive sampling, saving months of storage costs.

Future predictions for 2026 and beyond

NVLink Fusion and architecture diversification (e.g., RISC-V + NVLink) will increase NVLink topological variety; telemetry must be topology-aware (late 2025–2026 trend).
Hardware vendors will continue to expose richer link-level telemetry; expect NVLink-specific error codes and per-VLS (virtual link) stats to become standard in 2026.
OpenTelemetry and vendor collectors will converge on richer semantic conventions for GPU/NVLink labels — plan to migrate to these conventions to reduce custom parsing.
ML-powered root-cause analysis for multi-modal telemetry will move from research to standard infra tooling, reducing manual triage time further.

Checklist to get started in 90 days

Deploy DCGM exporter and OTEL collector as DaemonSets in your test cluster with ci.job.id injected into pods.
Enable per-link NVLink metrics collection and eBPF I/O tracing for per-process I/O visibility.
Create three Grafana dashboards: GPU overview, NVLink topology and throughput, and Storage I/O latency per test.job.id.
Define 3 alert rules for NVLink saturation, storage p99 regression, and GPU ECC errors with ci.job.id included in alert payloads.
Implement adaptive sampling: 1–5s resolution during active runs, and 30–60s otherwise; configure long-term rollups to object storage.

Final takeaways

Collect GPU, NVLink, and storage telemetry together. Enrich every data point with minimal contextual labels. Synchronize clocks tightly. Tier your storage and adapt sampling to test lifecycle. These patterns let teams detect and resolve performance regressions across ephemeral GPU fleets in minutes — and reduce the operational cost of observability.

Call to action

Ready to stop chasing regressions? Start with a 2-week pilot: deploy DCGM + OTEL DaemonSets, capture high-resolution telemetry for failing CI runs, and build the three dashboards from the 90-day checklist. If you want a templated starter kit (K8s manifests, OTEL configs, Grafana dashboards, and PromQL queries) tailored to your fleet size and cloud provider, contact our team at mytest.cloud for a free assessment.

mytest

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.