Review GPU usage and costs
Open GPUs in the sidebar for the full inventory, or select the GPUs tab on a cluster or virtual machine to narrow the view.- Filter by model, utilization, reporting time, or partitioning to find cards to review.
- Select a card. Overview shows its location, estimated cost, and any detected workloads or Multi-Instance GPU (MIG) partitions.
- Open Metrics for utilization, memory, power, and health charts. Use Usage to compare the latest measurements with 7-day and 30-day summaries.
Investigate idle or constrained GPUs
Use the utilization verdict to prioritize cards, then review usage history before changing capacity.
For partial or stale data, check the exporter and recent readings before
acting. CostGraph recommends changing a container’s GPU count only when its
GPUs are exclusive to that container.
Share whole GPUs with MIG slices
Enable per-process GPU attribution to identify workloads whose peak memory fits a smaller slice of their GPU. In Insights, open a GPU sharing recommendation to review the affected workloads, target slice, estimated savings, and partition-change plan. Recommendations also identify pods waiting for a slice missing from the node’s layout. Available recommendations depend on the GPU model and cloud. Changing the partition layout evicts every pod on the node. Upgrade the NVIDIA driver first if the plan requires it. On supported clouds, AutoPilot applies the plan after your team approves it. Refer to GPU partition changes for cloud support and approval requirements.Understand GPU cost estimates
GPU costs use catalog prices, not invoice totals.- Month-to-date cost estimates the observed hours in the current UTC month.
- Projected monthly cost estimates a full month at the current rate.
- Unpriced cards have no cost estimate and are excluded from the fleet spend total.
Set up GPU collection
Each GPU host needs the NVIDIA driver and DCGM exporter. Choose the setup for your environment, then return to GPUs to verify that cards report usage.Kubernetes
Enable the exporter and its scrape target in your operator values.device-name on GKE. For the standard NVIDIA device plugin, such as on
EKS, use uuid instead. The wrong value prevents CostGraph from linking cards
to their pods.
Apply the values with helm upgrade. See
Accelerators for profiling and per-process
attribution, or Bundled exporters
for selectors and tolerations.
Virtual machines and bare metal
Run the agent install script on a Linux GPU host. Whennvidia-smi is available and the host uses systemd, the script also
installs DCGM exporter. Rerun the script if the agent is already installed.
For an existing exporter, set COSTGRAPH_GPU_EXPORTER_ENDPOINT to its metrics
URL when running the installer. With Docker or without systemd, run the
exporter separately and set COSTGRAPH_DCGM_ENDPOINT on the agent.
Fix missing GPU data
Check the symptom to distinguish collection failures from unsupported metrics.GPUs or readings are missing
GPUs or readings are missing
Confirm
nvidia-smi lists the card and DCGM exporter is running on its host.
On Kubernetes, check both exporter settings and its pod’s placement. On a
systemd host, run systemctl status nvidia-dcgm-exporter.GPUs are not linked to pods
GPUs are not linked to pods
Check
DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE against your device plugin, then
apply the corrected operator values.A metric says Not reported by this card
A metric says Not reported by this card
The GPU or driver does not publish that metric. Other available metrics
remain usable. Refer to Measurement availability.