AI gateways

Most integrations read a bill. An AI gateway you run yourself has no bill to read: it measures the tokens it routes and reports them as metrics. CostGraph reads those metrics, prices them, and lands the result beside every other cost. The gateway can run on a Kubernetes cluster or on a virtual machine. This is its own way to connect, alongside pull, push, and exports:

CostGraph pulls it

You supply read-only credentials and we call the provider once a day.

You push it

You send us FOCUS records. Credentials never leave your infrastructure.

We read your telemetry

Your gateway already measures token usage. No credentials at all.

Supported gateways

Gateway	Attribute usage to your customers
Apache APISIX	Yes, already on every metric
Envoy AI Gateway	Yes, from a request header
Solo.io agentgateway	Yes, from any request expression
vLLM	Per workload only, see below

Each was run and measured rather than read about: every one reports tokens differently, and two of them do not follow the OpenTelemetry GenAI conventions at all. CostGraph normalises that for you. Each of the preceding pages lists the metrics we read, how their labels map, and the upstream metrics reference it was built against.

Connect

Open Settings > Integrations and pick your gateway.
Name the connection and pick the tenant this usage belongs to.
Under Where it runs, select A Kubernetes cluster or A virtual machine, then pick the cluster or the instance.
Select Connect.

Where it runs decides who scrapes it and how it’s located:

Where it runs	Scraped by	Located by
A Kubernetes cluster	The CostGraph operator in that cluster	Namespace and pod label selector
A virtual machine	The CostGraph agent on that host	A port on the machine’s loopback address

Pick Neither and the connection takes the usage you post to it instead. See Send usage yourself.

Kubernetes cluster

Enable the AI gateway scraper in your operator values once:

aiGatewayScraper:
  enabled: true

That is the whole cluster-side setup. You do not add scrape targets. CostGraph serves them from the connections you create, so connecting another gateway later changes nothing in your Helm values. A connection is picked up within five minutes of being created; the scraper refreshes its served targets on that interval rather than at start only. A served target always carries a metric-name filter, so it collects only the token metrics it was configured for. A target you set yourself in operatorPrometheus.scrapeTargets takes precedence over a served one.

agentgateway has no stock scrape location. Its token metrics come from the per-Gateway data plane, not the control plane, and those pods live in whichever namespace you created the Gateway in. So there is nothing for CostGraph to default to: an agentgateway connection must name the namespace and the pod label selector when you connect, or nothing can ever be scraped. Every other gateway defaults to its stock location, and you only supply these to override a non-standard deployment.

Pod selector

The pod selector is a comma-separated list of key=value terms, the same syntax kubectl get pods -l takes. Every term must match: the list is ANDed, so a pod is scraped only if it carries all of them.

app.kubernetes.io/name=agentgateway,gateway.networking.k8s.io/gateway-name=llm

One label is a valid list of one:

app.kubernetes.io/name=envoy-ai-gateway

Use equality terms only. !=, in (...), notin (...), and bare existence checks are not supported: CostGraph resolves the same selector to the pods it amortizes gateway cost across, and that resolution accepts key=value only. Match the labels the data plane pods actually carry, which you can read off the running pods:

kubectl -n my-namespace get pods --show-labels

Narrow the list until it selects the gateway’s data plane pods and nothing else - a selector that also matches other pods pulls their metrics into your gateway’s usage.

Virtual machine

A gateway on a virtual machine is scraped by the CostGraph agent on that machine. The agent reads the gateway’s metrics endpoint on the machine’s own loopback address, so nothing is exposed to the network and no credentials are collected. To connect one:

Under Where it runs, select A virtual machine.
Select the machine under Instance.
Confirm Metrics port, or change it if the gateway serves its metrics somewhere else.

The Instance list holds only machines whose agent is reporting, because an agent is what does the scraping. A machine that isn’t listed needs the agent installed on it first.

Default ports

CostGraph prefills Metrics port with the port the gateway serves metrics on in a stock install:

Gateway	Default port
Apache APISIX	`9091`
Envoy AI Gateway	`1064`
Solo.io agentgateway	`15020`
vLLM	`8000`

The prefill is a starting point, not a constraint. Running a gateway on another port is ordinary, so change the field to whatever yours serves on. Clearing the field falls back to the default. The port is the only thing that locates the gateway. The agent confirms it found the right process by checking the metric names the endpoint serves. You don’t name a process, a service, or a unit.

A wrong port

A port nothing serves fails quietly. The agent finds nothing to scrape, so no metrics arrive. The connection stays pending rather than reporting an error, and no surface says why. A connection sitting pending with no usage is the symptom, and the port is the first thing to check. Read the port off the running host before you rely on it:

curl -s http://127.0.0.1:8000/metrics | head

An empty or failed response is the same signal CostGraph gets.

Attribute usage to your customers

If you resell access, tell your gateway to label each request with your customer’s id. CostGraph reads that label into the FOCUS sub-account columns, so per-customer usage shows up wherever you already group by sub-account.

Envoy AI Gateway

Map a request header onto the metrics. This is a controller-wide setting, so every gateway that controller manages shares the mapping.

helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version v1.1.0 -n envoy-ai-gateway-system --reuse-values \
  --set controller.metricsRequestHeaderAttributes="x-costgraph-customer-id:customer.id"

The attribute customer.id arrives as the label customer_id. Dots become underscores on the Prometheus side.See Envoy AI Gateway.

Solo.io agentgateway

Add a metric attribute computed from the request. On Kubernetes this goes in an AgentgatewayPolicy. The value is a CEL expression, so the identifier can come from a header, a JWT claim, or anything else in the request.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: costgraph-customer-attribution
  namespace: my-gateway-namespace
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: my-gateway
  frontend:
    metrics:
      attributes:
        add:
          - name: customer
            expression: request.headers["x-costgraph-customer-id"]

The policy must live in the same namespace as the Gateway it targets.See Solo.io agentgateway.

Apache APISIX

Nothing to configure. Give each customer its own APISIX consumer; the consumer label is already on every LLM metric, because consumers are APISIX’s own auth identities.APISIX only starts reporting LLM metrics after the first AI request passes through it, so an empty scrape before any traffic is expected.See Apache APISIX.

vLLM

Not available per request. vLLM is a model server, not an auth boundary, and reports only the model name, so no configuration changes that.Either front it with a gateway that can attribute per request, or run one deployment per customer and annotate the workload:

kubectl annotate deployment/llama-serving \
  costgraph.ai/sub-account=acme-corp

See vLLM for the metrics we read and how a model you host yourself is priced.

The annotation fills in a customer only where the gateway did not report one itself. A gateway that labels each request wins, so annotating a shared gateway does not collapse its customers onto one sub-account. Annotate the workload only where per-request attribution is not available.

How usage is priced

Rates resolve in order, and the first match wins:

Your own rate, if the workload carries a costgraph.ai/pricing-id.model label. Register the rate once in the pricing marketplace and tag with the id it returns. See Pricing IDs.
A hosted reference rate for the same model, from the model catalog. This answers “what would this have cost on someone else’s API”.
Your own amortized rate, for a model you serve yourself. CostGraph divides what the serving namespace cost that day by the tokens it served that day, so a self-hosted model prices at what it actually cost you to run rather than at somebody else’s list price.

A pin that names a rate which no longer exists leaves the usage unpriced, and never silently falls back to a market rate. A plausible wrong number is worse than no number.

Only token usage prices today. Images, audio seconds and characters are captured and stored now, and will price once the catalog carries rates for them, so nothing is lost in the meantime.

Connection settings

Three fields tune how a connection’s usage is priced and attributed. You set model_map and attribution_spec when you connect the gateway, and tenant_map on a connection that already exists.

Model map

Your gateway reports whatever model string its callers asked for. model_map translates those raw identifiers into the keys the CostGraph model catalog prices. Usage that matched no rate starts pricing. Reach for it when a connection reports that models could not be priced. The message names the raw identifiers the gateway sent, and each one is a key you map:

{
  "model_map": {
    "gpt4o-prod": "openai/gpt-4o",
    "Qwen/Qwen2.5-0.5B-Instruct": "qwen/qwen2.5-0.5b-instruct"
  }
}

Keys match the reported identifier exactly, after trimming surrounding whitespace. A model with no entry resolves under the name it was reported with, and an entry mapping to an empty string is ignored. Map a model only where the catalog already knows the target key. For a model you host yourself that no catalog rate fits, leave it unmapped. Register your own rate instead, as described in the preceding rate order.

Tenant map

Turns raw sub-account values into the names you want to read:

{ "tenant_map": { "acct-7": "Acme" } }

This renames a value your gateway already reported. Usage the gateway left unlabelled stays unattributed, so a tenant map cannot assign spend to a customer on its own.

Attribution spec

Declares where each attribution dimension is read from. Every rule names the dimension, its source, and the key to read:

{
  "attribution_spec": {
    "sub_account_dimension": "customer",
    "dimensions": [
      { "name": "customer", "source": "prom_label", "key": "customer_id" },
      { "name": "team", "source": "prom_label", "key": "team" }
    ]
  }
}

source is prom_label for a gateway we scrape, focus_field for usage you push, or static for a fixed value carried in key. Adding "transform": "sha256" hashes the value before it is stored, for an identifier you would rather not keep in the clear. A rule marked "required": true fails the batch when its key is absent, rather than storing usage with the dimension missing. sub_account_dimension picks which dimension fills the FOCUS sub-account columns. The others ride along as cost extensions, so you can group by them alongside spend. Omit the spec and a connection still attributes usage: a single sub_account dimension reads costgraph_sub_account on a scraped gateway, and SubAccountId on a pushed row.

Send usage yourself

If your gateway runs outside a CostGraph-managed cluster, send the usage directly instead. The rows are FOCUS 1.2 shaped with the cost columns left empty, because CostGraph prices them:

curl -X POST https://api.costgraph.ai/api/v1/tenant/billing/focus/usage \
  -H "Authorization: Bearer $COSTGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "connection_id": "fpc_...",
    "focus_version": "1.2",
    "charge_period_start": "2026-08-20T00:00:00Z",
    "charge_period_end": "2026-08-21T00:00:00Z",
    "rows": [{
      "SkuId": "meta/llama-3.3-70b",
      "SkuMeter": "meta/llama-3.3-70b/output",
      "ConsumedQuantity": "20000",
      "ConsumedUnit": "tokens"
    }]
  }'

The key needs the focus:write scope.

A row that already carries BilledCost, EffectiveCost, ListCost or ContractedCost is rejected. If your gateway computes its own spend, use Push FOCUS data instead, which takes conformant records and stores them as they are.

Send from anything else

CostGraph reads one metric family. Anything that can write it can be a source - a Prometheus server, an OpenTelemetry Collector, or Grafana Alloy, whether or not it runs on Kubernetes:

costgraph_gen_ai_usage_total{
  costgraph_tenant_id="0d5f...",       # required
  costgraph_gateway="prod-inference",  # required
  costgraph_model="gpt-4o",            # required
  costgraph_usage_kind="input",        # required
  costgraph_unit="tokens",             # required
  costgraph_sub_account="acme",        # optional
  costgraph_host="openai",             # optional
  costgraph_cluster_id="...",          # optional
  costgraph_cluster_name="aws-prod",   # optional
}

The value is a monotonically increasing counter of units consumed.

The labels

Label	Required	What it is for
`costgraph_tenant_id`	Yes	Your CostGraph tenant. Together with `costgraph_gateway` this is the key that matches a series to a connection, so usage carrying neither is never claimed by anything.
`costgraph_gateway`	Yes	The connection this usage belongs to. It must equal the gateway name you set when you connected, so two gateways reporting the same model never merge.
`costgraph_model`	Yes	The model the call ran against. This is what resolves a rate, through the connection’s model map if you set one.
`costgraph_usage_kind`	Yes	Which side of the call the number counts, so each side is priced at its own rate rather than a blended one. One of `input`, `output`, `cached_input`, `cache_write`, `reasoning`, `image`, `audio_input`, `audio_output`, `embedding`. A gateway that names its cache buckets `input_cache_read` and `input_cache_write` is understood too - they are read as `cached_input` and `cache_write`. Anything else is dropped.
`costgraph_unit`	Yes	What the value counts: `tokens`, `images`, `seconds`, `characters`. Anything else is dropped.
`costgraph_sub_account`	No	Your customer, when you attribute usage. It lands in the FOCUS sub-account columns, so per-customer spend groups the way the rest of your bill does.
`costgraph_host`	No	Who served the call - `openai`, `anthropic`, your own cluster. Rate lookup prefers a price published for that host, so the same model resolves differently per host.
`costgraph_cluster_id`, `costgraph_cluster_name`	No	The cluster the gateway runs in. The operator stamps these on scraped series; a gateway that remote-writes from outside Kubernetes has nothing to put here.

A series missing any required label, carrying an unknown costgraph_usage_kind or costgraph_unit, or reporting a value of zero or less, is skipped rather than half-priced. Any other label rides along and is available to attribution dimensions, which read the labels your gateway already emits, such as team, route or gen_ai_provider_name, into sub-account and cost extensions. Nothing needs renaming to costgraph_ first. Remote-write it to https://tsdb.costgraph.ai/api/v1/write with an API key in the X-API-Key header. A gateway CostGraph has never heard of works this way with no changes on our side.

Get Started

CostGraph Operator

CostGraph Agent

GPUs

Insights

Alerts

Budgets & Forecast

Allocation

Virtual Tags

Network

Workflows

MCP

Slack

Security

Developers

Self-Hosted

Integrations

Changelog

CostGraph pulls it

You push it

We read your telemetry

Supported gateways

Connect

Kubernetes cluster

Pod selector

Virtual machine

Default ports

A wrong port

Attribute usage to your customers

How usage is priced

Connection settings

Model map

Tenant map

Attribution spec

Send usage yourself

Send from anything else

The labels

CostGraph pulls it

You push it

We read your telemetry

​Supported gateways

​Connect

​Kubernetes cluster

​Pod selector

​Virtual machine

​Default ports

​A wrong port

​Attribute usage to your customers

​How usage is priced

​Connection settings

​Model map

​Tenant map

​Attribution spec

​Send usage yourself

​Send from anything else

​The labels

Supported gateways

Connect

Kubernetes cluster

Pod selector

Virtual machine

Default ports

A wrong port

Attribute usage to your customers

How usage is priced

Connection settings

Model map

Tenant map

Attribution spec

Send usage yourself

Send from anything else

The labels