> ## Documentation Index
> Fetch the complete documentation index at: https://docs.costgraph.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CostGraph AutoPilot

> Keep Kubernetes workload CPU, memory and GPU requests right-sized automatically, without editing your manifests.

CostGraph AutoPilot keeps a workload's CPU and memory requests right-sized,
sizes its GPUs, and can move it onto the node pool that runs it cheapest. You
turn it on with one annotation on a workload or a namespace. CostGraph never
edits your Deployments, StatefulSets, DaemonSets, Jobs or CronJobs, so nothing
changes in Git.

> [!NOTE]
>
> CostGraph AutoPilot is free during preview.

Every managed workload shows a "Managed by CostGraph AutoPilot" chip that says
how it is managed, what changed and when, and what it has saved.

## Requirements

* The [CostGraph Operator](https://docs.costgraph.ai/costgraph/operator/overview.md) installed on the cluster.
* Kubernetes 1.28 or later for the admission webhook.
* Kubernetes 1.33 or later to resize running pods in place. On older clusters,
  sizing is applied when pods are created and the chip says so.
* A learning period. A workload is not changed until CostGraph has a confident
  recommendation for it, and the chip shows "Learning" until then.

## Turn it on

**1. Enable it on the cluster**

AutoPilot is off by default. Turn on the cluster switch and the way changes are applied:

```bash theme={null}
helm upgrade costgraph-operator costgraph/costgraph-operator \
  --reuse-values \
  --set global.autopilot.enabled=true \
  --set global.autopilot.webhook.enabled=true
```

See [Operator configuration](https://docs.costgraph.ai/costgraph/operator/configuration.md#costgraph-autopilot) for every setting.

**2. Annotate a namespace or a workload**

```bash theme={null}
kubectl annotate namespace payments autopilot.costgraph.ai/mode=initial
```

A workload annotation wins over its namespace. Start with `initial`, watch the
timeline, then move to `auto`.

## Modes

| `autopilot.costgraph.ai/mode` | What happens |
| - | - |
| `initial` | New pods start with the right-sized requests. Running pods are not touched. |
| `auto` | New pods start right-sized, and running pods are resized in place. |
| `off` | Nothing is applied. The recommendation is still shown. |
| `disabled` | The workload is excluded entirely. Use it to opt a workload out of an annotated namespace. |
| not set | Recommendations only. |

The annotation settings are listed in [Annotations](https://docs.costgraph.ai/costgraph/autopilot/annotations.md).

## How it applies changes

You choose how sizing is applied with `autopilot.costgraph.ai/via`, or cluster-wide with `global.autopilot.defaultVia`.

> **webhook (default)**
>
> CostGraph sets requests on pods as they are created and, with `mode: auto` on
> Kubernetes 1.33 or later, resizes running pods in place without evicting
> them, a few pods at a time. It also supports GPU sizing, pool placement,
> rollback after a failed change, and sizing of Jobs.

> **[vpa](https://docs.costgraph.ai/costgraph/autopilot/vpa.md)**
>
> CostGraph supplies recommendations to a Vertical Pod Autoscaler that it creates
> for the workload, and the VPA applies them.

A workload is never handled by both. If a workload is targeted by a
VerticalPodAutoscaler that CostGraph did not create, AutoPilot skips it. On the
VPA path the workload shows "Has own VPA".

## Sizing profiles

`autopilot.costgraph.ai/profile` sets how much headroom a request keeps over
observed usage.

| Profile | Target utilisation | Effect |
| - | - | - |
| `tight` | 90% | Smallest requests, most savings, least headroom. |
| `balanced` (default) | CostGraph recommendation | Headroom depends on how steady the workload is: 90% target for steady workloads, 60% for spiky ones, otherwise 80%. |
| `safe` | 60% | Largest requests, most headroom. |

A request is the usage CostGraph expects divided by the target utilisation, so a
`safe` request is larger than a `tight` one. Your own `cpu.min`, `cpu.max`,
`memory.min` and `memory.max` bounds are applied on top.

## What AutoPilot checks before every change

Changes are applied in this order, and each step can hold a change back. A held
change is still shown as a recommendation with the reason.

1. The profile sets the starting request.
2. Your minimum and maximum bounds are applied. If a minimum is higher than its maximum, both are ignored and the chip reports it.
3. The request is capped to what the workload's node pool can launch, after DaemonSet overhead and system reserves. A cap is never exceeded.
4. A resource an HPA scales on is dropped from control (see below).
5. Learning period, direction, minimum change (default 10%), minimum savings, cooldown and change window.
6. After an out-of-memory kill of a running pod, the memory request and limit are raised by 20% straight away, ignoring cooldown and window and never above your `memory.max` or the pool cap. The raise fades back toward the recommendation over three days. This applies to `mode: auto` workloads on the webhook path.
7. Limits keep their current ratio to the request. A limit is never added where there is none and never removed.

If a workload has no current recommendation, for example while a CronJob is between runs, AutoPilot reuses the last applied size for up to 7 days, then leaves new pods unchanged.

## Horizontal Pod Autoscalers

If an HPA scales a workload on CPU or memory utilisation, AutoPilot stops managing that resource, because changing the request would change how the HPA scales. The chip shows why.

* A utilisation metric on a resource (`Resource`) drops that resource for every container.
* A utilisation metric on one container (`ContainerResource`) drops that resource for that container only.
* Average-value, custom and external metrics drop nothing.

## Platform limits

* **Quality of service is preserved.** A Guaranteed pod keeps requests equal to limits. A Burstable pod never becomes Guaranteed, and a BestEffort pod is never given requests. Requests and limits are never removed.
* **Memory limits are not lowered in place.** They are lowered only when a pod is created. Resizing a running pod changes CPU and memory requests, and memory limits only grow.
* **Jobs and CronJobs are sized when they are created** and never resized or evicted while they run.
* **GPUs are changed at pod creation only**, never in place, and only with `gpu: auto`. GPU model steering also needs `placement: auto`, and happens only when a node or a CostGraph-owned node pool in the cluster carries that GPU model.
* **Running pods are resized only for CPU and memory of regular containers.** Init containers, GPUs and other extended resources are not resized. Windows pods are not resized in place.
* **Rollouts pause in-place resizes.** A running workload in the middle of a rollout is left alone until it finishes. Sizing of new pods continues.
* **In-place resizes are paced.** At most 10% of a workload's pods (at least one) are resized per cycle.
* **In-place resizes have a cluster-wide cap.** At most 50 pods per cluster are resized per cycle.
* **A failed change is rolled back.** If a pod is killed for running out of memory or starts crash-looping within 30 minutes of an in-place change, the change is reverted on the workload's pods and held for 30 minutes. Throttling is not used as a trigger.
* **A resize the node cannot take is not retried.** The same size is not sent again, and a new recommendation is tried on a later cycle. A resize blocked by a namespace quota is reported once rather than retried. A resize the node defers is waited on for a few cycles and then dropped.
* **Node savings from a downsize arrive when nodes consolidate.** Karpenter does not yet count in-place resizes when it consolidates, so node cost drops only once nodes are replaced. Savings are measured, not assumed.
* **Pods in `kube-system`, `kube-node-lease` and the CostGraph namespace are never sized or resized** by AutoPilot, and pods without a controller are never sized or resized.

## Kill switches

You can stop AutoPilot at three levels, from broadest to narrowest.

| Switch | Scope |
| - | - |
| `global.autopilot.paused=true` (Helm) | The whole cluster. Nothing is applied and new pods start unchanged. |
| `autopilot.costgraph.ai/paused-until` on a namespace | Every workload in that namespace, until the date passes. |
| `autopilot.costgraph.ai/mode: disabled`, or `paused-until`, on a workload | One workload. |

AutoPilot also fails open. If CostGraph is unreachable, or its data is older than
`global.autopilot.staleAfter` (30 minutes by default), pods start unchanged and
running pods are no longer resized.

## Node pools

With `placement: auto`, AutoPilot can steer a workload onto a node pool that runs it more cheaply.

* With `global.autopilot.ownedNodePools.enabled=true`, CostGraph can create its own Karpenter NodePools, labelled `costgraph.ai/managed: "true"`, when a split of an existing pool saves money. See [Karpenter](https://docs.costgraph.ai/costgraph/autopilot/karpenter.md).
* CostGraph never edits your own NodePools, node groups or compute classes. Suggested changes to them are opened as pull requests for you to review.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.