> ## Documentation Index
> Fetch the complete documentation index at: https://docs.costgraph.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# View gateway metrics

> Watch requests, latency, tokens, and spend per model for a connected AI gateway

The **Metrics** tab shows how a connected AI gateway is serving traffic. It charts
requests, latency, tokens, and spend per hour for each model, so you can relate
a change in cost to the traffic behind it.

The tab is available for gateways that CostGraph collects from a Kubernetes
cluster or a virtual machine. A connection that sends
[custom usage](https://docs.costgraph.ai/costgraph/integrations/ai-gateways/custom-usage.md) does not have a
**Metrics** tab. Its usage still appears in **Cost breakdown**.

## Open the Metrics tab

You can open the same charts from two places.

* Open **Integrations**, select the **Connected** tab, and select your gateway.
  Then select the **Metrics** tab next to the connection tabs.
* Open **Cost breakdown** and group by **Provider**. Select your gateway's row
  to open its provider page, then select the **Metrics** tab.

The charts show data for one gateway connection at a time. If the gateway has
more than one connection, choose the connection from the first menu in the
chart controls.

## Chart controls

The menus at the top of the tab change what every chart shows.

| Control | Options | Default |
| - | - | - |
| Connection | Each collected connection for this gateway. Appears only when there is more than one. | The first connection |
| Model | **All models** or one model the gateway reported in the range | **All models** |
| Latency quantile | **Median**, **95th percentile**, or **99th percentile** | **95th percentile** |
| Time range | From **Last 5 minutes** to **Last 30 days** | **Last 24 hours** |

Each chart draws one line per model. A model keeps the same color in every chart.
Model names appear as the gateway reports them. Select the refresh button on a
chart to load the latest data.

The following screenshot shows the charts for an agentgateway connection that
serves two models.

[![The Metrics tab showing Spend per hour, Requests per second, End-to-end latency, Tokens per request, Input tokens per second, and Output tokens per second charts with one line for each of two models.](https://mintcdn.com/baselinehq/stFYHxxbSi4CZfGv/images/ai-gateways/gateway-metrics.png?fit=max\&auto=format\&n=stFYHxxbSi4CZfGv\&q=85\&s=cc282d1581e70e8c36b92d364f836b85)](https://docs.costgraph.ai/images/ai-gateways/gateway-metrics.png)

Gateway metrics for two models using sample data. Select the image to view it at full size.

## Gateway charts

The tab shows the following charts.

| Chart | What it shows |
| - | - |
| Spend per hour | The token rate priced at the rate CostGraph charges for this connection's usage on the latest usage day, in US dollars per hour. It agrees with **Cost breakdown**. |
| Requests per second | The number of model requests the gateway completed each second. |
| End-to-end latency | The selected quantile of request duration, from the gateway receiving a request to finishing its response. The axis uses the unit the gateway reports. |
| Tokens per request | The average number of tokens in each request, counting all token types. |
| Input tokens per second | The rate of input tokens the gateway reported. |
| Output tokens per second | The rate of output tokens the gateway reported. |

The line next to the menus reports the most recent traffic, such as
**Last traffic 5 minutes ago**, or **No traffic in this range**. Its dot turns
amber when the gateway reported no tokens in the last 30 minutes of the range.
The tab then shows a warning with the time of the last traffic and a
**Check the connection** link.

A model without a rate has no **Spend per hour** line. The tab names those models
in a note at the top of the tab and counts no cost for their tokens, the same as the
bill. To price them, see [Model map](https://docs.costgraph.ai/costgraph/integrations/ai-gateways.md#model-map)
and [Use your own model rates](https://docs.costgraph.ai/costgraph/integrations/ai-gateways.md#use-your-own-model-rates).

## Metrics by gateway

Every supported gateway reports token usage, so the token and spend charts work
for all of them. The request, latency, and tokens per request charts need the
gateway to report request duration.

| Gateway | Token and spend charts | Requests, latency, and tokens per request |
| - | - | - |
| Apache APISIX | Yes | No |
| Envoy AI Gateway | Yes | Yes |
| agentgateway | Yes | Yes |
| LiteLLM | Yes | Yes |
| vLLM | Yes | Yes |

For a gateway that does not report a measurement, the chart says so instead of
showing zero. For example, an Apache APISIX connection shows
**This gateway is not reporting request latency.** on the latency chart.

## Connection alerts

CostGraph alerts you when a gateway connection stops delivering data. The alert
appears in several places.

* A banner at the top of every dashboard page names the connection, or counts
  the affected connections, with a link to them. Select the close button to hide
  it until the alerts change or you sign in again. The banner does not appear on
  the **Integrations** pages, which show each alert in full.
* A dot marks **Integrations** in the sidebar. Hover over it to list each
  connection that needs attention.
* A red badge marks the gateway in **Integrations > Connected**.
* The gateway's integration page and provider page show a banner for each
  affected connection.

Each alert describes one of the following conditions.

| Alert | Cause | What to do |
| - | - | - |
| *Connection* hasn't delivered any data | No usage has arrived since the connection was created more than two hours ago, or since the date in the message. | Send a model request through the gateway. Then check that the namespace, pod selector, and port match where the gateway runs. |
| *Connection* isn't delivering data | CostGraph read the gateway's metrics but could not use them. The message names the reason. | Fix the reason in the message, such as a missing label in your attribution settings. |
| *Connection* needs attention | Usage is arriving, but one or more models have no rate. The message names the models. | Map or price the models. Refer to [How usage is priced](https://docs.costgraph.ai/costgraph/integrations/ai-gateways.md#how-usage-is-priced). |

For a gateway that has not delivered data, the banner lists the two usual causes.
Either the gateway has not served a model request yet, or the collector cannot
reach it. Gateways publish token and latency metrics only after they serve a
request to a model.

The following screenshot shows the banner for a connection that has delivered no
data since it was created.

[![An error banner titled qa-agentgateway hasn't delivered any data, with the reason, the two usual causes, and a Check the connection link.](https://mintcdn.com/baselinehq/stFYHxxbSi4CZfGv/images/ai-gateways/gateway-connection-alert.png?fit=max\&auto=format\&n=stFYHxxbSi4CZfGv\&q=85\&s=05a540b67c4c5edb6179342bc2c684e2)](https://docs.costgraph.ai/images/ai-gateways/gateway-connection-alert.png)

A connection alert using sample data. Select the image to view it at full size.

## Empty charts

An empty chart means CostGraph received no data for that measurement in the
selected range. Check the following before you change the connection.

1. Widen the time range. A gateway that served no requests in the last hour has
   nothing to chart for that hour.
2. Choose **All models**. A model filter hides other models' traffic.
3. Read the chart's message. **The gateway served no requests in this range.**
   means data was expected but none arrived. **This gateway is not reporting
   request latency.** means the gateway does not publish that measurement. See
   [Metrics by gateway](#metrics-by-gateway).
4. Check that the gateway's metrics endpoint responds and reports token usage.

### Check the metrics endpoint

Forward the gateway's metrics port to your machine to read what it publishes.
This example uses agentgateway on its default port.

```bash theme={null}
kubectl -n my-gateway-namespace port-forward pod/my-gateway-pod 15020:15020
```

In another terminal, request the metrics and look for the token metric.

```bash theme={null}
curl --fail --silent --show-error http://localhost:15020/metrics \
  | grep '^agentgateway_gen_ai_client_token_usage'
```

On a virtual machine, run `curl` against the loopback address on the host instead.
Use the port and path for your gateway, and look for its metrics in the output.

| Gateway | Port | Path | Token metrics | Request duration metric |
| - | - | - | - | - |
| Apache APISIX | `9091` | `/apisix/prometheus/metrics` | `apisix_llm_prompt_tokens`, `apisix_llm_completion_tokens` | Not available |
| Envoy AI Gateway | `1064` | `/metrics` | `gen_ai_client_token_usage` | `gen_ai_server_request_duration_seconds` |
| agentgateway | `15020` | `/metrics` | `agentgateway_gen_ai_client_token_usage` | `agentgateway_gen_ai_server_request_duration` |
| LiteLLM | `4001` | `/metrics/` | `litellm_input_tokens_metric_total`, `litellm_output_tokens_metric_total` | `litellm_request_total_latency_metric` |
| vLLM | `8000` | `/metrics` | `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | `vllm:e2e_request_latency_seconds` |

If the endpoint does not respond, correct the port in the connection or in the
gateway's configuration. If it responds without token metrics, send a model
request through the gateway and check again. Each gateway's guide in
[Supported gateways](https://docs.costgraph.ai/costgraph/integrations/ai-gateways.md#supported-gateways)
describes the settings that publish its metrics.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.