Grafana - CostGraph Documentation

CostGraph queries your Grafana when you open a page and stores nothing. Your metrics stay where they are; the connection only lets CostGraph read them. This is different from the Grafana Cloud billing integration, which brings your Grafana Cloud invoice into CostGraph. Use that one for cost, and this one for host metrics.

What it covers

CostGraph reads metrics from Grafana for these pages:

Utilization and sparklines on the virtual machine list.
Metrics on a virtual machine’s detail page.
Top processes on a virtual machine’s detail page.

It does not change Kubernetes, GPU, container, anomaly, or recommendation pages. Those keep using the data your CostGraph agents and operators send. Only Macs running the CostGraph agent can read their metrics from Grafana, and CostGraph reads the agent’s own costgraph_* series. Each Mac declares where its metrics live in its agent configuration: a Mac reads from Grafana only if its configuration names this connection. A Mac that names no connection keeps using the metrics CostGraph already has for it, and every other host does the same.

Choose how to connect

Grafana Cloud metrics

You need the Prometheus URL and instance ID from your stack’s Prometheus Details page, plus an access policy token with the metrics:read scope.

A Grafana data source

You need your Grafana stack URL, the UID of the data source with your host metrics, and a service account token. The service account needs the Viewer role and permission to query that data source (datasources:query). If data source permissions are restricted, the Viewer role alone is not enough.

Pick the second option when your metrics live in a data source Grafana already reaches for you, including self-hosted and private data sources. CostGraph asks Grafana to run the query, so it works with Prometheus-compatible data sources only: Prometheus, Mimir, VictoriaMetrics, and Thanos. Other Grafana data source types are not supported.

Create the token

For Grafana Cloud metrics, create an access policy on grafana.com:

Sign in at grafana.com and open your organization’s Access Policies page, for example https://grafana.com/orgs/example-org/access-policies/new. Replace example-org with your organization’s slug.
Choose the realm that contains your stack.
Add the metrics:read scope and create the policy.
Add a token to the policy and copy it. Grafana does not show it again.

For A Grafana data source, create the service account inside your Grafana stack, not on grafana.com. Open Administration > Users and access > Service accounts. Give it the Viewer role, then add a token.

Find a data source UID

In Grafana, open Connections > Data sources.
Open the data source that holds your host metrics.
Copy the UID from the page address.

Connect

Open Settings > Integrations and pick Grafana.
Choose Grafana Cloud metrics or A Grafana data source.
Enter the address, the instance ID or data source UID, and the token.
Set the options in Label filters and Extra matchers.
Select Connect.
Copy the connection id from Integrations into each Mac’s agent configuration. See Point a Mac at this connection.

CostGraph runs a short query against your store before saving, so a wrong address, token, or label setting fails while you are still on the form. You can edit a connection later. Leave the token empty to keep the stored one, unless you change the address or how CostGraph signs in.

Point a Mac at this connection

Set these in the Mac’s agent configuration file, next to api_key:

remote_write: false
metrics_connection_id: "<connection-id>"
metrics_instance: "<instance-label>"

metrics_connection_id is the id of this connection, copied from Integrations.
remote_write: false stops the agent pushing host and process metrics to CostGraph. The Mac still registers, sends heartbeats, and shows up; its metrics are read from Grafana.
metrics_instance is optional. It is the instance label value your scrape produces for this Mac. Leave it out to use the hostname.

metrics_connection_id can be at most 64 characters and metrics_instance at most 255. Neither can contain spaces, quotes, backslashes or braces. A Mac that names a connection but has no series in it shows that there is no data from the connection. It pushes nothing to CostGraph, so CostGraph’s own metrics do not stand in. If Metrics is switched off on the connection, the Mac shows CostGraph’s own metrics, which are empty when remote_write is false. Switch Metrics back on or point the Mac at another connection. If the connection never had metrics on, the agent’s registration is refused with the message in the troubleshooting table. metrics_only: true cannot be combined with remote_write: false or metrics_connection_id. The agent refuses to start with that configuration: remove them or set metrics_only to false. metrics_instance only applies together with metrics_connection_id. After you connect, the dialog shows these settings ready to copy.

Use one connection for cost and metrics

One connection can serve more than one use. In Integrations, the Used for checkboxes Cost and Metrics turn each use on or off for the connection. A Mac can only name a connection that has Metrics switched on. To use an existing Grafana Cloud cost connection for metrics too, choose Also use for metrics. CostGraph reuses the token stored with that connection, so you enter no second token. Both need a token whose access policy has metrics:read, as in Create the token. Otherwise add the scope to the policy, or reconnect with a token that has it.

Set the instance label

Scrape the CostGraph agent on each Mac and set instance to the bare hostname, or to the value you put in metrics_instance. You can copy these snippets before you connect. With Prometheus:

scrape_configs:
  - job_name: costgraph-agent
    static_configs:
      - targets: ["<mac-hostname>:9101"]
    relabel_configs:
      - source_labels: [__address__]
        regex: "(.+?)(?::[0-9]+)?"
        target_label: instance
        replacement: "$1"

With Alloy:

discovery.relabel "costgraph_agent" {
  targets = [{"__address__" = "<mac-hostname>:9101"}]
  rule {
    source_labels = ["__address__"]
    regex         = "(.+?)(?::[0-9]+)?"
    target_label  = "instance"
    replacement   = "$1"
  }
}

prometheus.scrape "costgraph_agent" {
  targets    = discovery.relabel.costgraph_agent.output
  forward_to = [prometheus.remote_write.default.receiver]
}

Replace <mac-hostname> with each Mac’s hostname, or list all your Macs as targets. A Mac matches when the instance label equals its hostname, with or without :port and with or without .local. Only the .local suffix is matched without regard to case; hostnames are otherwise case-sensitive. If you scrape by IP address or by a DNS name that is not the Mac’s hostname, set metrics_instance in the agent file to that exact instance value.

Label filters

Use Label filters when your store holds series for more than one environment or workspace. Each filter is a label name and a value, and CostGraph adds all of them to every query. For example, the filter env = production reads only series labelled env="production".

Add at most 10 filters.
A label name uses letters, digits, and underscores, and does not start with a digit.
A value cannot be empty and can be at most 256 characters.
You cannot filter on instance, tenant_id, or any label that starts with __ or costgraph_.

Extra matchers

Use Extra matchers for filters that a label name and value cannot express. Enter optional PromQL-style label matchers, for example region=~"eu-.*". CostGraph adds them to every query after the label filters, so a series must satisfy all of them. Extra matchers can be at most 1024 characters and cannot select a metric name. CostGraph refuses a query that it cannot tie to your workspace’s hosts, instead of running it. A refused query shows as an error on the page. Macs that share the same instance value in your store cannot be told apart. Give each Mac a different value.

Charts on Grafana

Charts that your series support fill in from Grafana: CPU, memory, load, disk space, network, and top processes. Some charts stay empty for macOS hosts: pressure, disk I/O, steal, and iowait. macOS does not report these, so Grafana has nothing to return. Top processes are computed from the raw process series, so you do not need recording rules in your store.

Security

The token is stored encrypted and is not shown again.
Only public https addresses are allowed. An address with a username or password in it is refused; put the token in the token field.
Queries are read-only. CostGraph does not write to your Grafana.
CostGraph queries your Grafana only for your own workspace’s hosts.

Check the connection

Open Settings > Integrations and check the Grafana connection shows no error.
Open a Mac that names this connection. Its metrics load from Grafana.
If Grafana stops answering, the connection shows an error and affected pages report that metrics could not be read. Fix the cause and open a page again.

Troubleshoot errors

Message	What to do
Enter your Grafana address, for example https://your-stack.grafana.net	The address is empty or malformed. Enter the full `https` address.
This Grafana address points to a private or internal network. CostGraph can only reach public Grafana addresses.	Use the public address of your Grafana.
This Grafana address could not be found. Check the address.	Check the address for typos and that its DNS name resolves publicly.
Enter the Grafana token again when you change the address or how CostGraph signs in	Paste the token again when you edit the address or switch connection option.
Put the token in the token field, not in the address	Remove the username and password from the address and paste the token in the token field.
Grafana must be reached over https	Use an `https` address.
Enter the metrics instance ID shown on your Grafana Cloud stack’s Prometheus details	Copy the instance ID from the stack’s Prometheus Details page.
Enter the UID of the Grafana data source that holds your host metrics	Follow Find a data source UID.
Choose Grafana Cloud metrics or a Grafana data source	Pick one of the two connection options.
Enter the Grafana token	Paste the access policy token or service account token.
Label filters: use at most 10 label filters	Remove filters until 10 or fewer remain.
Label filters: “name” is not a valid label name; use letters, digits and underscores, starting with a letter or underscore, at most 128 characters	Rename the label.
Label filters: the label “name” is reserved and cannot be used as a filter	Remove filters on `instance`, `tenant_id`, and labels starting with `__` or `costgraph_`.
Label filters: the label filter “name” needs a value	Enter a value, or remove the filter.
Label filters: the label filter “name” can be at most 256 characters	Shorten the value.
The extra matchers can be at most 1024 characters	Shorten the extra matchers.
The extra matchers are not valid label matchers, for example env=“production”	Write each matcher as `label="value"`, `label=~"regex"`, and so on, separated by commas.
CostGraph can only reach Grafana on port 443. Remove the port from the address.	Remove the port from the address.
Extra matchers cannot use instance, tenant_id, costgraph_ or __ labels; CostGraph sets those itself	Remove matchers on those labels.
Grafana is already connected for this workspace. Edit or remove that connection first.	Edit the existing connection instead of adding another.
Grafana did not accept CostGraph’s query. Check the address, the token and that the token can read metrics.	Check the address. For Grafana Cloud metrics, confirm the token has the `metrics:read` scope. For a data source, confirm the service account has the Viewer role and the UID is right.
Grafana answered but host metrics could not be read. Check the label filters and extra matchers.	Review your label filters and extra matchers.
Metrics are already set up on this connection. Edit them instead.	Edit the existing metrics settings of the connection.
Set up the Grafana connection for metrics first.	Connect Grafana for metrics before you switch Metrics on for a connection.
No Grafana or Prometheus connection with this id has metrics enabled. Copy the id from Integrations and check that metrics is switched on.	Copy the connection id from Integrations into `metrics_connection_id` and switch Metrics on for that connection.
This Grafana Cloud connection cannot be used for metrics. Reconnect it with a token that can read metrics.	Create a token whose access policy has the `metrics:read` scope, then reconnect. See Create the token.
N hosts use this connection and have no metrics in it yet. Check that their instance label is their hostname, with or without .local.	Shown as a warning on the connection; it reads “1 host uses this connection and has no metrics in it yet. Check that its instance label is its hostname, with or without .local.” for a single host. Compare the hosts’ `instance` labels with the values in your store, and set the label as in Set the instance label or set `metrics_instance`.
N hosts use this connection for metrics but it is not set up for metrics. Turn metrics on for it or point those hosts at another connection.	Shown as a warning on the connection; it reads “1 host uses this connection for metrics but it is not set up for metrics. Turn metrics on for it or point that host at another connection.” for a single host. Switch Metrics on for the connection, or change `metrics_connection_id` on those hosts.
Grafana did not answer CostGraph’s queries. Check the address and token on Integrations.	The token may be revoked or expired, or the address changed. Edit the connection with a new token.
Metrics from Grafana could not be read. Check the Grafana connection on Integrations.	Shown on a virtual machine page. Open the connection and fix the address or token.

​What it covers

​Choose how to connect

Grafana Cloud metrics

A Grafana data source

​Create the token

​Find a data source UID

​Connect

​Point a Mac at this connection

​Use one connection for cost and metrics

​Set the instance label

​Label filters

​Extra matchers

​Charts on Grafana

​Security

​Check the connection

​Troubleshoot errors