Request header attribution on macOS - CostGraph Documentation

On a Mac, the CostGraph Agent can read the X-CostGraph-<key> headers your services send and attribute each request’s bytes to their values. A Mac reports the same attribution_<key> tags as a Linux host. Each request is attributed to the process that sent it. The feature is off by default. Header names become tag keys by the same rules on every host. See How header names become keys.

Decide whether you need this

Most Macs do not need request header attribution. The agent already reports each Mac’s network flows without any setup: the destination, the bytes in each direction, and the process that opened the connection. For Linux hosts and Kubernetes, header attribution works with no proxy. Turn this feature on only when services that run on the Mac send X-CostGraph-* headers and you need their bytes split by those values. It has a cost:

HTTPS requests are intercepted, so clients must trust the agent’s CA.
Programs that pin certificates fail through the proxy.
Go programs and other clients that verify through the keychain fail until the CA is in the keychain.
Only requests that go through the proxy are attributed.

If you accept these costs, the rest of this page sets it up.

How it works

A Linux host reads headers with eBPF. A Mac uses the attribution proxy, a local forwarding proxy that the agent runs on a loopback port. Requests that clients send through the proxy are attributed. Requests that do not use it are not. For an HTTPS request, the proxy terminates TLS with a certificate authority (CA) that the agent creates on the Mac. It then opens its own TLS connection to the destination. The client must trust that CA, as described in Trust the CA. The proxy keeps only the values of X-CostGraph-* request headers. It does not keep or send any other header, path, query string, or body. The CA private key stays on the Mac. The agent matches each request to the process that opened the connection. Requests to Apple services and certificate authorities, such as apple.com, icloud.com, and letsencrypt.org, pass through the proxy without interception.

Requirements

The agent is installed on the Mac with Homebrew and registers with CostGraph. See the macOS tab in Installation.
metrics_only is not true. Attribution needs a registered agent.
Network flow tracing is on. network_trace defaults to true in the configuration file.
Clients send their requests through the proxy and trust its CA.
For HTTPS, clients offer application-layer protocol negotiation (ALPN) in their TLS handshake. curl and browsers do.

Enable attribution

Add an http_attribution block to the agent configuration file at $(brew --prefix)/etc/costgraph_agent/config.yaml. The following example listens on a fixed port. The agent exports the CA files that clients read to /usr/local/share/costgraph/ unless you set other paths.

http_attribution:
  enabled: true
  listen_port: 8888

Restart the service to apply the change.

sudo brew services restart costgraph-agent

The agent refuses to start when enabled: true is combined with metrics_only: true. It also refuses a CA path that is not absolute, and an upstream other than auto, none, or a host:port.

boolean

default:"false"

Turns attribution on. When you set it back to false, the agent removes its CA, the exported CA files, and any keychain trust it added. It also restores the system proxy.

number

default:"0"

Loopback port the proxy listens on at 127.0.0.1. 0 picks a free port each time the agent starts. Set a fixed port when you configure clients to use the proxy yourself.

boolean

default:"false"

Points the Mac’s web and secure web proxy settings at the attribution proxy. See Use the system proxy.

string

default:"auto"

Where the proxy sends requests. auto forwards through the web proxy the Mac already has configured, or connects directly when there is none. none connects directly. A host:port value forwards through that HTTP proxy. An upstream proxy that also intercepts TLS breaks HTTPS through the attribution proxy.

boolean

default:"false"

Adds the agent’s CA to the System keychain. See Trust the CA.

string

Absolute path where the agent writes the public CA certificate. The file is readable by every user. Defaults to /usr/local/share/costgraph/attribution-ca.pem.

string

Absolute path where the agent writes the macOS system root certificates followed by the agent’s CA. The file is readable by every user. Defaults to /usr/local/share/costgraph/attribution-bundle.pem.

Point clients at the proxy

A client uses the proxy when its own proxy setting names it. Many clients read the HTTPS_PROXY and HTTP_PROXY environment variables.

export HTTPS_PROXY=http://127.0.0.1:8888
export HTTP_PROXY=http://127.0.0.1:8888

Use the system proxy

With system_proxy: true, the agent sets the web and secure web proxy of the Mac’s active network service to the attribution proxy after the proxy is healthy. Clients that follow the macOS proxy settings then use it without further configuration.

Every client that follows the system proxy goes through interception. A client that does not trust the agent’s CA fails every HTTPS request. Turn on system_proxy only on a Mac where those clients trust the CA.

The agent records the previous settings and restores them exactly when it stops, when the proxy goes down, and after a crash. Restoring also clears a proxy host and port that the agent set. If the proxy keeps failing, the agent restarts it with a growing delay and then retries every five minutes. The system proxy stays restored until the proxy is healthy again. The agent does not apply the system proxy in these cases:

The network service uses an automatic proxy configuration (PAC) file.
The network service already uses a different proxy and upstream is none. With auto, the existing proxy becomes the upstream instead.

Trust the CA

The CA matters only for HTTPS. Requests over plain HTTP are not intercepted, so they are attributed with no CA at all. For HTTPS, a client that does not trust the CA fails with a certificate error. macOS does not let the agent trust its CA by itself, because an administrator trust change needs interactive approval. Give clients the CA in one of these ways:

Point clients at the exported files (ca_cert_path and ca_bundle_path, by default under /usr/local/share/costgraph/). This works for clients that read a certificate file.
Set trust_system_keychain: true only on a Mac where the trust change can be approved, such as one managed by mobile device management (MDM). If macOS refuses the change, attribution stays off.
Deliver the certificate from ca_cert_path to the System keychain with your MDM. Clients that verify through the keychain need this.

The agent writes both files each time it starts. The CA is valid for one year. The agent replaces it when it starts within 30 days of expiry. When you turn attribution off, the agent deletes both files. If the macOS system roots at /etc/ssl/cert.pem are missing or empty, the agent does not write the bundle and attribution stays off. Set the variable each client reads.

Variable	Points at	Effect
`SSL_CERT_FILE`	The bundle	Replaces the default certificate list.
`REQUESTS_CA_BUNDLE`	The bundle	Replaces the default certificate list.
`CURL_CA_BUNDLE`	The bundle	Replaces the default certificate list.
`NODE_EXTRA_CA_CERTS`	The CA certificate	Adds the CA to the default list.

The first three variables replace the client’s default list. The bundle includes the system roots, so the client still verifies servers that the proxy does not intercept.

export SSL_CERT_FILE=/usr/local/share/costgraph/attribution-bundle.pem
export REQUESTS_CA_BUNDLE=/usr/local/share/costgraph/attribution-bundle.pem
export CURL_CA_BUNDLE=/usr/local/share/costgraph/attribution-bundle.pem
export NODE_EXTRA_CA_CERTS=/usr/local/share/costgraph/attribution-ca.pem

Client support

The following results were measured through the attribution proxy.

Client	Trust setting	Result
curl from macOS (`/usr/bin/curl`)	System keychain	Trusted and attributed once the CA is in the System keychain.
curl from Homebrew or another build that uses its own certificate list	`CURL_CA_BUNDLE`	Trusted and attributed.
`nscurl`	System keychain	Trusted and attributed once the CA is in the System keychain.
Python 3.12 or later, `urllib`	`SSL_CERT_FILE`	Trusted and attributed.
Python `requests`	`REQUESTS_CA_BUNDLE` or `CURL_CA_BUNDLE`	Trusted and attributed. `requests` ignores `SSL_CERT_FILE`.
Python `urllib3`	`SSL_CERT_FILE`	Trusted and attributed.
Node.js	`NODE_EXTRA_CA_CERTS`	Trusted and attributed.
Clients that send no ALPN, such as the macOS system Python 3.9	None	Passed through unchanged and not attributed.
Go programs, such as `gh`, `terraform`, and `kubectl`	System keychain	TLS fails unless the CA is trusted in the System keychain.
Safari, Xcode, and apps that use `URLSession`	System keychain	TLS fails unless the CA is trusted in the System keychain.

Go programs on macOS ignore SSL_CERT_FILE and verify through the keychain. Keep Go tools off the proxy on a Mac where the keychain does not trust the CA. Python 3.13 and later verify certificates strictly, and the agent’s CA meets those checks.

Verify attribution

Check that the proxy is up. The agent serves its local metrics on port 9101.

curl -s http://localhost:9101/metrics | grep costgraph_attribution_proxy_state

A value of 1 means the proxy is up. Then send a tagged request through the proxy.

curl --proxy http://127.0.0.1:8888 \
  --cacert /usr/local/share/costgraph/attribution-ca.pem \
  -H "X-CostGraph-Tenant: acme" \
  -o /dev/null -w '%{http_code}\n' https://github.com/

The command prints 200. The proxy counts the request in costgraph_attribution_proxy_records_total{outcome="received"} on the same local endpoint.

Read the metrics

The agent ships two request series to CostGraph. They are not served on the local /metrics endpoint.

Series	Value
`costgraph_flowtrace_http_req_bytes_total`	Bytes the client sent.
`costgraph_flowtrace_http_resp_bytes_total`	Bytes the client received.

Each sample holds the bytes counted since the previous collection. Both series carry these labels:

attribution_<key> for each X-CostGraph-<key> header on the request
server_address, the host the request was sent to
name and effective_name, the process that sent the request
src_ip, dst_ip, and dst_port of the outbound connection

The tags are exact. The byte counts are estimates built from the decoded request and response, which includes the request line, headers, and body. Measured against curl, response bytes are within about 0.01 percent. Request bytes leave out headers that the proxy adds when it forwards the request. HTTP/2 sizes are decoded sizes, not compressed bytes on the wire. The agent also reports the health of the proxy.

Metric	Meaning
`costgraph_attribution_proxy_state`	`0` stopped, `1` up, or `2` degraded after repeated failures.
`costgraph_attribution_proxy_restarts_total`	Number of proxy restarts.
`costgraph_attribution_proxy_records_total`	Request records by `outcome`, which is `received`, `dropped_full`, or `invalid`.
`costgraph_flowtrace_http_spans_dropped_total`	Requests not fully attributed, by `reason`.

The reason label on dropped requests has three values:

unowned means no process was found for the request. It is published without name.
cap-exceeded means a limit was reached. See Limits.
malformed means the request record was invalid and was dropped.

Limits

8 tag keys per request and 256 bytes per value, as on Linux.
64 server addresses per process. Requests to further hosts are published without server_address.
256 distinct tag sets per process in each collection. Requests with further tag sets are published without tags.
Interception covers only clients that offer ALPN. Other HTTPS clients pass through unattributed.
Clients that do not send requests through the proxy are not attributed.
Tart virtual machine guests on the Mac are not covered.

Troubleshoot attribution

When attribution cannot start, the agent keeps running without it and logs a line that starts with http attribution is off, followed by the reason. Read it in $(brew --prefix)/var/log/costgraph_agent.log or $(brew --prefix)/var/log/costgraph_agent.error.log.

Reason in the log	Fix
`trust the interception CA`	macOS refused the keychain trust change. Set `trust_system_keychain` to `false` and use the exported files, or approve the change through MDM.
`write the CA bundle`	The system roots at `/etc/ssl/cert.pem` are missing or empty. Restore them. The attribution proxy stays off until the file is valid.
`state dir`	The `attribution` directory in the agent’s state directory is not owned by the user the agent runs as. The state directory is `/Library/Application Support/costgraph-agent` by default. Make root its owner.
`uses an automatic proxy configuration`	The network service uses a PAC file. Turn off the automatic proxy configuration, or set `system_proxy` to `false`.
`already uses the proxy`	Another proxy is set and `upstream` is `none`. Set `upstream` to `auto` to forward through it, or set `system_proxy` to `false`.

With network_trace: false, the agent instead logs that HTTP attribution needs network trace and stays off. Set network_trace to true. The following problems appear in the client rather than in the agent log.

Symptom	Cause	Fix
HTTPS requests fail with a certificate error.	The client does not trust the agent’s CA.	Set the variable the client reads, as described in Trust the CA.
Requests succeed but are not attributed.	The client sends no ALPN, so the proxy passes its connection through.	Use a client that offers ALPN.
A Go tool such as `gh` or `terraform` fails TLS.	Go programs on macOS verify only through the keychain.	Trust the CA in the System keychain, or keep the tool off the proxy.

​Decide whether you need this

​How it works

​Requirements

​Enable attribution

​Point clients at the proxy

​Use the system proxy

​Trust the CA

​Client support

​Verify attribution

​Read the metrics

​Limits

​Troubleshoot attribution

Decide whether you need this

How it works

Requirements

Enable attribution

Point clients at the proxy

Use the system proxy

Trust the CA

Client support

Verify attribution

Read the metrics

Limits

Troubleshoot attribution