Skip to main content
CostGraph AutoPilot keeps a workload’s CPU and memory requests right-sized, sizes its GPUs, and can move it onto the node pool that runs it cheapest. You turn it on with one annotation on a workload or a namespace. CostGraph never edits your Deployments, StatefulSets, DaemonSets, Jobs or CronJobs, so nothing changes in Git.
CostGraph AutoPilot is free during preview.
Every managed workload shows a “Managed by CostGraph AutoPilot” chip that says how it is managed, what changed and when, and what it has saved.

​
Requirements

  • The CostGraph Operator installed on the cluster.
  • Kubernetes 1.28 or later for the admission webhook.
  • Kubernetes 1.33 or later to resize running pods in place. On older clusters, sizing is applied when pods are created and the chip says so.
  • A learning period. A workload is not changed until CostGraph has a confident recommendation for it, and the chip shows “Learning” until then.

​
Turn it on

1

Enable it on the cluster

AutoPilot is off by default. Turn on the cluster switch and the way changes are applied:
See Operator configuration for every setting.
2

Annotate a namespace or a workload

A workload annotation wins over its namespace. Start with initial, watch the timeline, then move to auto.

​
Modes

The annotation settings are listed in Annotations.

​
How it applies changes

You choose how sizing is applied with autopilot.costgraph.ai/via, or cluster-wide with global.autopilot.defaultVia.

webhook (default)

CostGraph sets requests on pods as they are created and, with mode: auto on Kubernetes 1.33 or later, resizes running pods in place without evicting them, a few pods at a time. It also supports GPU sizing, pool placement, rollback after a failed change, and sizing of Jobs.

vpa

CostGraph supplies recommendations to a Vertical Pod Autoscaler that it creates for the workload, and the VPA applies them.
A workload is never handled by both. If a workload is targeted by a VerticalPodAutoscaler that CostGraph did not create, AutoPilot skips it. On the VPA path the workload shows “Has own VPA”.

​
Sizing profiles

autopilot.costgraph.ai/profile sets how much headroom a request keeps over observed usage. A request is the usage CostGraph expects divided by the target utilisation, so a safe request is larger than a tight one. Your own cpu.min, cpu.max, memory.min and memory.max bounds are applied on top.

​
What AutoPilot checks before every change

Changes are applied in this order, and each step can hold a change back. A held change is still shown as a recommendation with the reason.
  1. The profile sets the starting request.
  2. Your minimum and maximum bounds are applied. If a minimum is higher than its maximum, both are ignored and the chip reports it.
  3. The request is capped to what the workload’s node pool can launch, after DaemonSet overhead and system reserves. A cap is never exceeded.
  4. A resource an HPA scales on is dropped from control (see below).
  5. Learning period, direction, minimum change (default 10%), minimum savings, cooldown and change window.
  6. After an out-of-memory kill of a running pod, the memory request and limit are raised by 20% straight away, ignoring cooldown and window and never above your memory.max or the pool cap. The raise fades back toward the recommendation over three days. This applies to mode: auto workloads on the webhook path.
  7. Limits keep their current ratio to the request. A limit is never added where there is none and never removed.
If a workload has no current recommendation, for example while a CronJob is between runs, AutoPilot reuses the last applied size for up to 7 days, then leaves new pods unchanged.

​
Horizontal Pod Autoscalers

If an HPA scales a workload on CPU or memory utilisation, AutoPilot stops managing that resource, because changing the request would change how the HPA scales. The chip shows why.
  • A utilisation metric on a resource (Resource) drops that resource for every container.
  • A utilisation metric on one container (ContainerResource) drops that resource for that container only.
  • Average-value, custom and external metrics drop nothing.

​
Platform limits

  • Quality of service is preserved. A Guaranteed pod keeps requests equal to limits. A Burstable pod never becomes Guaranteed, and a BestEffort pod is never given requests. Requests and limits are never removed.
  • Memory limits are not lowered in place. They are lowered only when a pod is created. Resizing a running pod changes CPU and memory requests, and memory limits only grow.
  • Jobs and CronJobs are sized when they are created and never resized or evicted while they run.
  • GPUs are changed at pod creation only, never in place, and only with gpu: auto. GPU model steering also needs placement: auto, and happens only when a node or a CostGraph-owned node pool in the cluster carries that GPU model.
  • Running pods are resized only for CPU and memory of regular containers. Init containers, GPUs and other extended resources are not resized. Windows pods are not resized in place.
  • Rollouts pause in-place resizes. A running workload in the middle of a rollout is left alone until it finishes. Sizing of new pods continues.
  • In-place resizes are paced. At most 10% of a workload’s pods (at least one) are resized per cycle.
  • In-place resizes have a cluster-wide cap. At most 50 pods per cluster are resized per cycle.
  • A failed change is rolled back. If a pod is killed for running out of memory or starts crash-looping within 30 minutes of an in-place change, the change is reverted on the workload’s pods and held for 30 minutes. Throttling is not used as a trigger.
  • A resize the node cannot take is not retried. The same size is not sent again, and a new recommendation is tried on a later cycle. A resize blocked by a namespace quota is reported once rather than retried. A resize the node defers is waited on for a few cycles and then dropped.
  • Node savings from a downsize arrive when nodes consolidate. Karpenter does not yet count in-place resizes when it consolidates, so node cost drops only once nodes are replaced. Savings are measured, not assumed.
  • Pods in kube-system, kube-node-lease and the CostGraph namespace are never sized or resized by AutoPilot, and pods without a controller are never sized or resized.

​
Kill switches

You can stop AutoPilot at three levels, from broadest to narrowest. AutoPilot also fails open. If CostGraph is unreachable, or its data is older than global.autopilot.staleAfter (30 minutes by default), pods start unchanged and running pods are no longer resized.

​
Node pools

With placement: auto, AutoPilot can steer a workload onto a node pool that runs it more cheaply.
  • With global.autopilot.ownedNodePools.enabled=true, CostGraph can create its own Karpenter NodePools, labelled costgraph.ai/managed: "true", when a split of an existing pool saves money. See Karpenter.
  • CostGraph never edits your own NodePools, node groups or compute classes. Suggested changes to them are opened as pull requests for you to review.