Requirements
A request is attributed only when all of the following are true:- The instrumented process is the HTTP client. Attribution is counted on the side that sends the request. A server that receives tagged requests does not attribute them.
- The request carries at least one non-empty
X-CostGraph-<key>header. Requests without one are not attributed. - The protocol is HTTP/1.1 or HTTP/2 over TCP. HTTP/3 (QUIC over UDP) is not supported.
- The TLS stack is supported. This means Go’s
crypto/tls, or a dynamically loadedlibssl.sothat exportsSSL_readandSSL_write. See Runtime compatibility. - The client runs on a Linux node the agent monitors and is visible through
the host’s
/proc. - The process resolves to a container. Requests from processes running directly on the host are dropped unless host-flow collection is enabled.
- Every proxy hop you want to attribute preserves the header. See Proxies and sidecars.
Runtime compatibility
The rule behind the table: CostGraph needs a TLS library it can hook. Node, Bun, Deno, and static or musl builds compile their TLS into the binary.
rustls, Java JSSE, and BoringSSL are TLS stacks CostGraph doesn’t hook yet.
To check whether a binary links OpenSSL dynamically, run
ldd /path/to/binary | grep libssl. A libssl.so line in the output means the
library can be hooked.
Enable capture
Per-request capture runs in the network metrics scraper. Where you turn it on depends on how you run CostGraph:- Operator
- Agent
In the operator chart’s values, set The
perRequestCollector on the flowtrace
component:flowtrace DaemonSet already mounts the host’s /proc at /host/proc, so it
can see client processes. See
Network Metrics scraper
for its other settings.Host processes
By default, requests from processes that don’t belong to a container are dropped. To attribute traffic from processes running directly on the node, also enable host-flow collection:flowtrace.config.includeHostFlows: true in the
operator chart, or --flowtrace-include-host-flows on the agent.
Send tagged requests
Add one or moreX-CostGraph-<key> headers to the outgoing requests you want to
attribute. The part after X-CostGraph- is the tag key, and the header value is
the tag value.
- Go
- Python
- curl
How header names become keys
Because dashes become underscores,
X-CostGraph-Cost-Center and
X-CostGraph-Cost_Center produce the same key.
Limits
- 8 tag keys per request. Extra keys are not recorded.
- 256 bytes per value. Longer values are truncated. Invalid UTF-8 is replaced with the Unicode replacement character.
Proxies and sidecars
Attribution happens at the client process that sends the request. Behind a proxy, each hop that is itself a supported client can be attributed. This works only if the header is still on the request it forwards.- If your gateway or proxy strips unknown headers, allow
X-CostGraph-*through so later hops can attribute too. - A sidecar that originates TLS (for example Envoy in Istio, or the Linkerd proxy) uses a TLS stack that isn’t supported yet. The application’s own request to the sidecar can still be attributed if the application sends it through a supported client.
Read the results
Open Network in the dashboard and go to Request header attribution. The cards at the top show totals for the selected period:- Request bytes and Response bytes are the total size of attributed requests and responses, including headers and bodies.
- Attributed cost is the network cost of those bytes, priced by where the traffic went.
Group and filter
- Group by picks one or more tag keys. With All tags, every key is shown.
A request with two keys, like
tenantandfeature, appears under both, so totals across different keys can add up to more than the overall total. - Filter by narrows the results to a tag value, a destination Port, or a Protocol.
- The table can be sorted by cost or bytes. Expand a row to see the connections behind it.
Locality
Each connection is priced by where its destination was:
A cost of
0 means the traffic was measured and the provider doesn’t bill it.
A - means the traffic couldn’t be priced because CostGraph doesn’t know where
the peer is with enough precision.
Common problems
My requests don't appear
My requests don't appear
Check each item in Requirements, in this order:
- The header is set on the outgoing request from the client, not on the response or on the server.
- The client’s runtime is in the supported half of the compatibility table.
- Per-request capture is on for that node. For the operator,
flowtrace.config.perRequestCollectordefaults tofalse. See Enable capture. - The process runs in a container, or host-flow collection is on.
- The key contains only letters, digits, dashes, or underscores, and the value is not empty.
Only some hops are attributed
Only some hops are attributed
A proxy along the path is probably stripping the header, or the hop uses an
unsupported TLS stack such as a BoringSSL or rustls sidecar. See
Proxies and sidecars.
Two keys I expected to be separate were merged
Two keys I expected to be separate were merged
Dashes become underscores, so
X-CostGraph-Cost-Center and
X-CostGraph-Cost_Center share the key cost_center. Rename one of them.Cost shows as -
Cost shows as -
The traffic was measured, but CostGraph couldn’t place the destination well
enough to price it. This is usually a private address that doesn’t match a
known resource, or a peer in the same network whose zone isn’t known.