Accelerators - CostGraph Documentation

The operator uses NVIDIA’s DCGM exporter to collect GPU utilization, memory, power, and health metrics. Enable it to monitor GPU nodes in CostGraph.

Enable GPU metrics

The exporter and its scrape target are disabled by default. Add both settings to your operator values file. The chart targets GPU nodes by default.

dcgm-exporter:
  enabled: true
operatorPrometheus:
  scrapeTargets:
    dcgm-exporter:
      enabled: true

Apply these settings with your existing release values. If your GPU nodes need custom selectors or tolerations, see Bundled exporters.

What is measured

CostGraph shows utilization, framebuffer memory, and power, plus the health and activity signals your hardware supports. These include temperature, XID errors, clock speeds, and encoder, decoder, and memory-copy activity. Energy charts show joules, converted from DCGM’s millijoules. PCIe byte readings cover the device’s sampling window. They are not cumulative counters.

How an unmeasured card is reported

Missing readings have a state instead of a zero value. Use the state to decide whether to check collection or the device’s capabilities.

State	Meaning
`supported`	The measurement is available.
`capacity_unknown`	A raw value is available, but its capacity is missing, so CostGraph cannot calculate a percentage.
`unsupported`	The card reports other measurements, but this signal is unavailable.
`no_data`	The card is not reporting. Check exporter scheduling and collection.

Available signals depend on the GPU, driver, and exporter configuration.

Opt-in profiling counters

SM active, SM occupancy, and PCIe byte readings require a separate profiling exporter. Enable it and its scrape target in your operator values.

dcgmProfiling:
  enabled: true
operatorPrometheus:
  scrapeTargets:
    dcgm-profiling:
      enabled: true

Some devices or drivers refuse profiling. Their profiling pods can fail while the base exporter continues collecting utilization, memory, and power. Use dcgmProfiling.nodeSelector to limit profiling to compatible nodes.

Per-process GPU attribution

The GPU exporter attributes each card’s memory and SM usage to the pod and container running on it. It runs as a DaemonSet on accelerator nodes, serves its metrics on port 9405, and is off by default. The image is registry.costgraph.ai/costgraph-gpu-exporter. Pulling it needs a CostGraph API key with the deployment:register scope. Store the key as an image pull secret in the operator’s namespace:

kubectl -n <operator-namespace> create secret docker-registry costgraph-registry \
  --docker-server=registry.costgraph.ai \
  --docker-username=x \
  --docker-password="$COSTGRAPH_REGISTRY_KEY"

Then turn on the exporter and its scrape target, and name the secret in the chart’s top-level imagePullSecrets:

imagePullSecrets:
  - name: costgraph-registry
gpuExporter:
  enabled: true
  driverInstallDir: /home/kubernetes/bin/nvidia
operatorPrometheus:
  scrapeTargets:
    gpu-exporter:
      enabled: true

driverInstallDir is where the NVIDIA driver lives on the host. The default is the GKE path. Set it to your driver directory on any other distribution. On a partitioned card, a process is attributed to the partition it runs in.

MIG-partitioned cards

Multi-Instance GPU (MIG) partitions use SM active and SM occupancy for activity measurements. Overall GPU, encoder, decoder, and memory-copy utilization are unavailable on partitioned cards. Power, energy, and temperature describe the whole card and are counted once. Per-workload attribution depends on the namespace, pod, and container labels available for each partition. Kubernetes GPU requests use whole devices or MIG partitions. Rightsizing changes the device count or partition layout.

Accelerator cost

GPU costs use catalog rates. Month-to-date cost covers elapsed hours in the current UTC month, starting when CostGraph first sees the card. Projected monthly cost estimates a full month at the current rates. These are estimates rather than invoice totals. Check pricing coverage before using a fleet total. A device without a catalog rate has no cost estimate.

​Enable GPU metrics

​What is measured

​How an unmeasured card is reported

​Opt-in profiling counters

​Per-process GPU attribution

​MIG-partitioned cards

​Accelerator cost