AutoPilot for sandboxes - CostGraph Documentation

AutoPilot for sandboxes chooses which of your sandbox providers starts each new sandbox. It also records when each sandbox starts and stops. You can then see what your sandboxes cost by provider and by the tags you set, such as agent, run, or customer. It works with ComputeSDK, the open source TypeScript SDK with one API for many sandbox providers. You wrap the ComputeSDK providers you already use with @costgraph/computesdk, and every other ComputeSDK call stays the same. From other languages, call the REST API directly. You keep your own provider accounts and keys. Your sandboxes run in those accounts, and each provider bills you as before. Your provider keys stay in your process and are never sent to CostGraph.

Requirements

A CostGraph API key with the sandbox:route scope. Create it in Settings > Personal API Keys.
Node.js 18 or later, or another runtime with a global fetch.
computesdk 4.1.15 or later, and the ComputeSDK provider package for each provider you use, such as @computesdk/e2b.

Set it up

Store your API key

Keep the key with the sandbox:route scope in your environment as COSTGRAPH_API_KEY.

Install the package

Install @costgraph/computesdk next to ComputeSDK.

npm install @costgraph/computesdk computesdk

Wrap your providers

Move your ComputeSDK providers inside costgraph(). The order of the list matters. The first provider is the one your sandboxes use when AutoPilot is off, and AutoPilot compares every other provider with it.

import { compute } from 'computesdk';
import { modal } from '@computesdk/modal';
import { vercel } from '@computesdk/vercel';
import { costgraph } from '@costgraph/computesdk';

compute.setConfig({
  provider: costgraph({
    apiKey: process.env.COSTGRAPH_API_KEY,
    providers: [
      modal({ tokenId: MODAL_TOKEN_ID, tokenSecret: MODAL_TOKEN_SECRET }),
      vercel({ token: VERCEL_TOKEN, teamId: VERCEL_TEAM_ID, projectId: VERCEL_PROJECT_ID }),
    ],
  }),
});

Creating, using, and destroying sandboxes stays the same.

Tag your sandboxes

Pass tags in metadata when you create a sandbox. CostGraph groups cost by these tags on the AutoPilot page.

const sandbox = await compute.instances.create({
  sizeClass: 'md',
  metadata: { agent: 'researcher', run: runId, customer: customerId },
});

Providers that pass metadata on to the vendor keep doing so.

Check that sandboxes arrive

Create a sandbox, then open AutoPilot in CostGraph and select the Sandboxes tab. The sandbox appears under History with its provider, size, and tags.

At this point CostGraph tracks what every sandbox costs, and your sandboxes still start on the first provider in your list. To let AutoPilot choose the provider, turn routing on as described in Turn on routing.

Flush reports on shutdown

The package sends start and stop reports in the background, and its timers do not keep your process running. Before your process exits, wait for queued reports with flush(). It resolves within 10 seconds by default, or within the limit you pass in milliseconds.

const cg = costgraph({
  apiKey: process.env.COSTGRAPH_API_KEY,
  providers: [e2b({ apiKey: process.env.E2B_API_KEY })],
});
compute.setConfig({ provider: cg });

process.on('SIGTERM', async () => {
  await cg.flush(5_000);
  process.exit(0);
});

On edge runtimes such as Cloudflare Workers, background work stops when the response is sent. Hand the flush to the runtime with ctx.waitUntil(cg.flush()).

Package options

costgraph() accepts the following options.

Option	Default	Description
`apiKey`	Required	Your CostGraph API key. If it is empty, sandboxes start in list order and nothing is reported.
`providers`	Required	Your ComputeSDK providers. Each must have a unique `name`.
`baseUrl`	`https://api.costgraph.ai`	The CostGraph API address.
`tenantId`	None	The tenant to report to, sent as the `X-CostGraph-Tenant-ID` header.
`routeTimeoutMs`	`300`	The longest a create waits for a routing answer, in milliseconds.
`cacheTtlMs`	`300000`	How long a routing answer is reused, in milliseconds.
`fetch`	Global `fetch`	A custom `fetch` implementation.
`onError`	Ignore	A function called with any CostGraph error, for logging. Errors never reach your sandbox calls.
`fallbackOnError`	`true`	Set to `false` to never try another provider after a failed create. ComputeSDK’s own `fallbackOnError` setting does not reach a wrapped provider, so set it here.

On a self-hosted installation, set baseUrl to your installation’s API address. Provider prices and start times reach your installation with its pricing updates.

Turn on routing

AutoPilot for sandboxes is off by default. While it is off, every sandbox starts on the first provider in your list, and CostGraph still records what each sandbox costs.

Open the settings

In CostGraph, open AutoPilot, select the Sandboxes tab, and select Settings. Only admins can change these settings.

Turn it on

Turn on Enabled.

Choose a routing policy

Select a Routing policy. The default is Cheapest. The policies are described in Policies.

New sandboxes pick up a change within a few minutes, because the package reuses each routing answer for up to five minutes.

How a provider is chosen

When you create a sandbox with a sizeClass, the package asks CostGraph to order your providers for that size. It then creates the sandbox on the first provider in that order.

Your first provider is the baseline. Every policy compares the other providers with it.
Only providers that support the size take part. If none of your providers supports the size, create() throws the same InstanceSizeError that ComputeSDK throws without CostGraph. Refer to Sizes.
A sandbox without a size stays on your first provider. Each provider has its own default size, so their prices are not comparable. CostGraph still records the sandbox and estimates its cost.
Your order is kept when AutoPilot can’t compare. This happens when AutoPilot is off, when the sandbox has no size, and when your first provider has no list price for the size.
Only the first 20 providers are priced. A provider name CostGraph does not recognize stays in the order without a price.

Expected cost and start time

AutoPilot compares providers on two numbers. Expected cost is what one sandbox of the requested size costs for its expected run time. The run time is rounded up to the provider’s billing increment, and is at least its minimum billed time, at the provider’s list price per hour. The expected run time is the median run time of your stopped sandboxes of that size that started in the last 30 days, capped at 24 hours. Until you have 20 such sandboxes, AutoPilot uses 10 minutes. Start time is the median time from the create request to the first command in the sandbox, in ComputeSDK’s public burst benchmark, which starts 100 sandboxes at once. The success rate is the share of those 100 starts that succeeded. A provider that the benchmark does not cover has no start time and no success rate.

Policies

Each policy orders your providers differently.

Policy	Order
Cheapest	Providers with a price for the size come first, lowest expected cost first. A provider whose start time is more than twice the baseline’s goes after the other priced providers. A missing start time never holds a provider back.
Balanced	Among providers that cost no more than the baseline and start within a limit, the best combination of saving and start time comes first. The baseline stays first when no provider improves on it.
Fastest	Among providers that cost no more than the baseline, the fastest to start comes first. The baseline stays first unless another provider starts strictly faster.

Under Balanced, the limit is the baseline’s start time plus 100ms or 10%, whichever is larger. Each provider within the limit scores its saving on the baseline’s expected cost multiplied by how far its start time is under the limit. Providers that score above zero come first, highest score first. Every policy applies the following rules:

A provider other than the baseline whose success rate is below 95% moves to the end.
Providers without a price for the size come after the priced providers.
Remaining ties keep your order.

The REST API page describes the full tie-break order for each policy.

When CostGraph doesn’t answer

Problems reaching CostGraph never make a sandbox create fail. A create waits up to routeTimeoutMs, which is 300ms by default. If CostGraph is slow, unreachable, or returns an error, the sandbox starts in the order of your providers list. An answer that arrives after the timeout is kept for the next create. After a failed call, creates use your list order for 30 seconds before asking again. The package reuses an answer for cacheTtlMs, which is 5 minutes by default, for each combination of size and provider list. Most creates make no extra call.

When a provider fails

If the chosen provider fails to create the sandbox, the package tries the next provider only when ComputeSDK would. It does not try another provider in the following cases:

The create has a sizeClass.
The create was aborted.
The provider marks the error as not safe to retry.
fallbackOnError is false.

Otherwise the package tries the next provider in CostGraph’s order, then any of your providers that the answer did not list. A create from a snapshot, or with volumes that CostGraph saw created or listed, goes to the provider that holds them.

Choose the provider yourself

To skip routing for one sandbox, call the costgraph() provider directly and name the provider. CostGraph still records the sandbox’s start and stop.

const sandbox = await cg.instances.create({
  provider: 'e2b',
  metadata: { agent: 'researcher' },
});

compute.instances.create({ provider: 'e2b' }) does not work for this, because compute knows only the costgraph provider.

Sizes

sizeClass requests one of ComputeSDK’s standard sizes.

Size	vCPUs	Memory
`xs`	1	1 GiB
`sm`	1	2 GiB
`md`	2	4 GiB
`lg`	4	8 GiB
`xl`	8	16 GiB
`2xl`	16	32 GiB

A provider supports a size in one of two ways:

Native. The ComputeSDK provider package maps sizes itself, as Archil and Namespace do. The package passes the size on unchanged.
Mapped. The provider package does not map sizes, but the provider accepts CPU and memory for each new sandbox. The package turns the size into that provider’s own create options, following the provider’s documented values.

The following providers take part in sized creates through mapping.

Provider	Sizes	Create options
Beam	`xs` to `xl`	`cpu`, `memory`
Daytona	`xs` to `lg`, only with `image`	`resources.cpu`, `resources.memory`
Modal	`xs` to `2xl`	`cpu` and `cpuLimit` in physical cores of two vCPUs each, `memoryMiB`
Northflank	`xs` to `2xl`	`deploymentPlan`, from `nf-compute-100-1` to `nf-compute-1600-32`
Runloop	`xs` to `2xl`	`launch_parameters.resource_size_request`, with a custom size from `lg` up
Vercel	`xs` to `xl`	`resources.vcpus`

Some providers have limits on the sizes they create:

Runloop and Vercel have no 1 vCPU, 1 GiB size. For xs they create their smallest size, 1 vCPU with 2 GiB. They are used for xs only when none of your other providers creates exactly 1 vCPU with 1 GiB, and CostGraph records the size they created.
Daytona accepts CPU and memory only for sandboxes built from an image. Pass image, such as image: 'python:3.12', together with sizeClass. Without image, or with snapshotId or templateId, Daytona does not take part in sized creates. xl and 2xl are above Daytona’s default limit of 4 vCPUs and 8 GiB.
Vercel allows 16 vCPUs only on Enterprise plans. 2xl is not sent to Vercel.

ComputeSDK does not allow CPU or memory options such as cpu, memory, resources, launch_parameters, or deploymentPlan next to sizeClass. Other options you pass are kept.

E2B templates

E2B sets CPU and memory per template, not per sandbox, so E2B does not take part in creates with sizeClass. To run E2B sandboxes at a given size, build a template with that size with E2B’s SDK. Then create sandboxes from it by name, without sizeClass.

await Template.build(template, 'agent-md', { cpuCount: 2, memoryMB: 4096 });

const sandbox = await cg.instances.create({
  provider: 'e2b',
  templateId: 'agent-md',
  metadata: { agent: 'researcher' },
});

CostGraph records these sandboxes without a size, so they are not routed or compared by size.

Sandboxes without a size

CostGraph prices a sandbox created without a size at its provider’s documented default size, when the provider’s price list gives one. History labels it Default size. A sandbox without a size is never moved to another provider.

What CostGraph receives

The package sends only the following to CostGraph:

Your CostGraph API key.
The names of your providers, such as e2b and daytona.
The size class of each create, or the vCPUs and memory a provider received when it created a larger size than requested.
For each sandbox, the provider name, sandbox ID, start and stop times, and the ID of the routing answer.
The metadata you pass to create(). Only string values are sent, with at most 20 keys. Keys can be up to 64 bytes, and values are cut to 256 bytes.

CostGraph never receives your provider keys, code, commands, environment variables, files, or command output. The package sends reports in batches of up to 100, at least every 2 seconds. It retries a failed send a few times and then drops it. It queues at most 10,000 reports, and drops the oldest beyond that.

Read the Sandboxes tab

Open AutoPilot and select the Sandboxes tab. The date range at the top of the page sets the period. Until CostGraph receives its first sandbox, Managed and History show the setup steps. The tab opens with the following totals for the period.

Tile	Meaning
Sandboxes	Sandboxes reported starting in the period.
Estimated cost	Each sandbox’s run time, up to 24 hours, rounded up the way its provider bills, at the list price for its size.
Saved	What routed sandboxes would have cost on the first provider in your list, minus what they cost.
Routed	Sandboxes AutoPilot started on a different provider than the first one in your list.
Running	Sandboxes with no stop reported yet. Their cost grows until they stop, up to 24 hours.
Unpriced	Sandboxes with no list price for their provider or size. They add no cost.

A sandbox counts toward Saved only when it started on the provider AutoPilot chose, at the size it was routed for. Its saving is measured against the first provider’s own price, billing increment, and minimum billed time. The tab has three views:

Managed shows Cost by provider, with the sandboxes, run time, and estimated cost on each provider. Below it, Cost by tag lists the 50 tags with the highest estimated cost, from the metadata you pass when you create a sandbox.
Settings has the Enabled switch and the Routing policy.
History lists every sandbox reported in the period, newest first, with when it started, its provider, size, run time, estimated cost, and tags. A routed sandbox shows the provider it was routed from and what it would have cost there. A sandbox with no stop reported shows Running.

Limits

A sandbox that ends without destroy() is not reported as stopped. This includes a sandbox that the provider’s own timeout stops. It shows as Running, and its estimated cost grows until the 24-hour cap. Call destroy() on every sandbox you finish with.
Run time counts at most 24 hours. Estimated cost covers at most 24 hours of each sandbox, including a sandbox that is still running.
Costs are estimates from list prices. They do not include discounts, credits, or included usage on your provider account. Your provider’s bill is what you pay. To see billed sandbox spend, connect the provider on the Integrations page where CostGraph supports it.
Prices and start times come from public sources. Expected cost uses each provider’s list price, and start time uses ComputeSDK’s public benchmark. Your own sandboxes can start faster or slower.

​Requirements

​Set it up

​Flush reports on shutdown

​Package options

​Turn on routing

​How a provider is chosen

​Expected cost and start time

​Policies

​When CostGraph doesn’t answer

​When a provider fails

​Choose the provider yourself

​Sizes

​E2B templates

​Sandboxes without a size

​What CostGraph receives

​Read the Sandboxes tab

​Limits

​Related