Skip to main content
LiteLLM reports input and output tokens as separate counters from its Prometheus callback, labelled with the virtual key, team, and end user of every request. See the upstream Prometheus metrics reference.

​
What CostGraph scrapes

Port 4001 is LiteLLM’s separate metrics server. The proxy’s own /metrics on port 4000 rejects a scrape without a virtual key, so turn the metrics server on wherever the proxy runs. The metrics server has no authentication. Keep it off public ingress.

​
On Kubernetes

Enable it in the litellm-helm chart:
values.yaml

​
On a virtual machine

Start the proxy with a metrics port:
The CostGraph agent on the machine reads port 4001 on the loopback address. The metrics port listens on every interface by default, so block 4001 at your firewall. Connect the gateway with A virtual machine under Where it runs. See Virtual machine.

​
Turn on the metrics

Add the Prometheus callback to the proxy config:
values.yaml
enable_end_user_cost_tracking_prometheus_only puts the end user on the metrics, which is how CostGraph attributes usage to your customers. The excluded labels are not used by CostGraph and would otherwise leave your cluster with the scrape.

​
How the metrics map

Each becomes costgraph_gen_ai_usage_total, with the direction set by which metric the value came from: The unit is always tokens. model is the upstream model LiteLLM called, not the alias from your proxy config, so each call is priced at the rate of the provider that served it. LiteLLM’s input count already includes cached and cache-write tokens, for every provider. CostGraph takes those out of input before pricing, so each token is billed once at its own rate. Request latency, queue time and time to first token feed the serving performance view. LiteLLM reports time to first token for streaming requests only.
A response served from LiteLLM’s own response cache still reports its tokens, so CostGraph prices it as if the provider had served it.

​
Attribute usage to your customers

Pass your customer’s id as the user field on each request. LiteLLM records it as the end user:
A request with no user is reported as end_user="None" and stays unattributed.

​
Avoid counting usage twice

LiteLLM can also be connected as a billing integration, which reads its spend logs once a day. Connect one or the other for the same proxy, not both. The same applies to the providers behind the proxy. If LiteLLM routes to OpenAI or Anthropic and you also connect that provider’s bill, the requests that went through LiteLLM appear in both.

AI gateways

Connecting, pricing, and sending usage yourself.