> ## Documentation Index
> Fetch the complete documentation index at: https://docs.costgraph.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run CI jobs on your own cluster

> Let AutoPilot for CI route GitHub Actions jobs to runner sets in your own Kubernetes cluster, priced at the share of the node each job uses.

> [!WARNING]
>
> Preview. Runner sets on your own cluster are not routed yet. This page
> describes how they work after the feature ships, so you can prepare a
> cluster ahead of time.

[AutoPilot for CI](https://docs.costgraph.ai/costgraph/autopilot/ci.md) can send a job to a runner set in
your own Kubernetes cluster, next to GitHub-hosted runners and runner providers.
A runner set is a group of self-hosted runners that GitHub's
[actions-runner-controller](https://docs.github.com/en/actions/concepts/runners/actions-runner-controller) creates.
It starts one short-lived runner pod per job, scales between its minimum and
maximum number of runners, and removes each runner when its job finishes.
CostGraph does not run or change the runners; it reads them and decides when a
job goes there.

## What CostGraph does

* **Finds your runner sets.** The [CostGraph operator](https://docs.costgraph.ai/costgraph/operator/overview.md)
  already running in the cluster reports every runner set, its name, the
  GitHub organization or repository it serves, the CPU and memory each runner
  requests, and whether Docker is available. No extra setup in CostGraph.
* **Offers them as runners when routing.** A runner set's name is the runner
  label AutoPilot returns. A running runner set is a candidate for every job of
  the organization or repository it is registered for.
* **Prices each job at the node's share.** A job costs the node's hourly price
  times the larger of its share of the node's CPU and its share of the node's
  memory. When a node has enough unused capacity for the runner, that capacity
  is already paid for, so the job counts as \$0 for routing and savings. The
  node's share is still recorded and shown next to every run, so you can see
  what the cluster spent.

Each routed job's cost comes from the node its runner pod actually ran on.

## Create a GitHub App for actions-runner-controller

actions-runner-controller needs its own GitHub credential to register runners.
It must be one you own: CostGraph's GitHub App isn't used, because its private
key would have to live in your cluster. We recommend a GitHub App over a
personal access token. A runner set registered for a GitHub enterprise can't
use a GitHub App; use a personal access token for it. GitHub's
[authentication guide](https://docs.github.com/en/actions/how-tos/manage-runners/use-actions-runner-controller/authenticate-to-the-api) has the details.

**1. Create the app**

In GitHub, go to your organization's **Settings > Developer settings >
GitHub Apps** and click **New GitHub App**. Clear **Active** under
**Webhook**; the app needs no webhook.

**2. Set its permissions**

For runner sets registered for the organization, set **Organization
permissions > Self-hosted runners** to **Read and write** and **Repository
permissions > Metadata** to **Read-only**. For runner sets registered for
one repository, set **Repository permissions > Administration** to **Read
and write** and **Metadata** to **Read-only** instead.

**3. Install it on the organization**

On the app's page, click **Install App** and install it on your
organization, for all repositories or the ones you select.

**4. Note the IDs and generate a private key**

The **App ID** is on the app's **General** page. The installation ID is the
number at the end of the installation's URL,
`https://github.com/organizations/<org>/settings/installations/<installation ID>`.
On the **General** page, click **Generate a private key** and save the
`.pem` file.

**5. Create the Kubernetes secret**

Create the secret in the namespace the runner set is installed in. With the
CostGraph operator chart, that's the release namespace, `costgraph`. If you
install actions-runner-controller yourself, use your runner set's namespace
instead.

```shell theme={null}
kubectl create secret generic arc-github-app \
  --namespace costgraph \
  --from-literal=github_app_id=123456 \
  --from-literal=github_app_installation_id=654321 \
  --from-file=github_app_private_key=path/to/private-key.pem
```

Set `githubConfigSecret: arc-github-app` in the runner set's values, as the
operator chart example in
[Install actions-runner-controller](#install-actions-runner-controller)
shows.

### Or use a personal access token

Create a personal access token (classic) with the `admin:org` scope for an
organization runner set, or the `repo` scope for a repository runner set. Store
it in the same secret as `github_token`:

```shell theme={null}
kubectl create secret generic arc-github-app \
  --namespace costgraph \
  --from-literal=github_token=YOUR_TOKEN
```

## Install actions-runner-controller

Use either of these.

> [tab] Install it yourself
>
> Follow GitHub's
> [quickstart](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/get-started)
> and [deploy a runner scale set](https://docs.github.com/en/actions/how-tos/manage-runners/use-actions-runner-controller/deploy-runner-scale-sets).
> Install it before or after the CostGraph operator; the operator finds the
> runner sets either way.

> [tab] With the CostGraph operator chart
>
> Set `arc.enabled: true` in the operator chart's values to install
> actions-runner-controller 0.15.1 and one runner set with the same release.
> The `arcController` and `arcScaleSet` values are passed unchanged to
> GitHub's controller and runner scale set charts, so every option in
> GitHub's documentation applies.
>
> The chart names the controller's service account `costgraph-arc-controller`
> and expects the release in the `costgraph` namespace. If you install the
> operator in another namespace, set
> `arcScaleSet.controllerServiceAccount.namespace` to it.
>
> ```yaml theme={null}
> arc:
>   enabled: true
> arcScaleSet:
>   githubConfigUrl: https://github.com/acme
>   githubConfigSecret: arc-github-app
>   runnerScaleSetName: acme-k8s-4cpu
>   minRunners: 0
>   maxRunners: 10
>   template:
>     spec:
>       containers:
>         - name: runner
>           image: ghcr.io/actions/actions-runner:latest
>           command: ["/home/runner/run.sh"]
>           resources:
>             requests:
>               cpu: "4"
>               memory: 16Gi
> ```
>
> `arc-github-app` is the secret from
> [Create a GitHub App](#create-a-github-app-for-actions-runner-controller).
> The runner set runs in Docker-in-Docker mode by default
> (`containerMode.type: dind`). For more than one runner set, install
> GitHub's runner scale set chart once per extra set.
>
> Helm installs the controller's custom resource definitions only on a fresh
> install. To turn on `arc.enabled` in an existing release, apply them before
> `helm upgrade`:
>
> ```shell theme={null}
> helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
>   --version 0.15.1 --untar
> kubectl apply --server-side -f gha-runner-scale-set-controller/crds
> ```
>
> Before you uninstall the release or turn `arc.enabled` off, delete the
> runner set first, so the controller can remove its runners from GitHub:
>
> ```shell theme={null}
> kubectl delete autoscalingrunnerset --all --namespace costgraph
> ```
>
> Helm leaves the actions-runner-controller resource definitions in the
> cluster. To remove them after you uninstall, run:
>
> ```shell theme={null}
> kubectl delete crd autoscalingrunnersets.actions.github.com \
>   autoscalinglisteners.actions.github.com \
>   ephemeralrunnersets.actions.github.com \
>   ephemeralrunners.actions.github.com
> ```

Set CPU and memory requests on the `runner` container. AutoPilot matches jobs
on them; without requests it uses what the job asks for.

### Docker

Jobs that need Docker go only to runner sets in Docker-in-Docker mode
(`containerMode.type: dind`, the operator chart's default). Runner sets without it, including the
`kubernetes` container mode, get no Docker jobs. See GitHub's
[runner scale set guide](https://docs.github.com/en/actions/how-tos/manage-runners/use-actions-runner-controller/deploy-runner-scale-sets)
for the modes.

## How routing picks it

A runner set competes with every other runner under the same rules:

* The workflow's [policy](https://docs.costgraph.ai/costgraph/autopilot/ci.md#policies) (`cheapest`,
  `fastest` or `balanced`) applies unchanged.
* The first time a job could run there, it runs once as a
  [trial](https://docs.costgraph.ai/costgraph/autopilot/ci.md#trials). The trial fails when it costs more
  than the default runner, or takes longer than the default runner's time plus
  the larger of 5 seconds and 10%. A failed trial is skipped for that job until
  its price is lower than the price it had at the trial.
* [Settings](https://docs.costgraph.ai/costgraph/autopilot/ci-settings.md) and
  [provider limits](https://docs.costgraph.ai/costgraph/autopilot/ci-settings.md#provider-limits) apply,
  with all runner sets under the `kubernetes` provider. Set
  `allow_secrets: false` there to keep jobs that use secrets off your cluster.
* If no runner picks a routed job up in time, for example because the runner
  set is at its maximum, AutoPilot moves the run back to its default runners
  (see [Stuck jobs](https://docs.costgraph.ai/costgraph/autopilot/ci.md#stuck-jobs)).

A runner set that is not running is skipped with the reason "runner set on
your cluster is not running".

## Limits

* **Enterprise runner sets.** A runner set registered for a GitHub enterprise
  can be routed to for every organization of that enterprise you connect to
  CostGraph.
* **Public repositories.** Runner sets join the **Default** runner group unless
  you set another. The Default group does not allow public repositories, so
  jobs from a public repository don't reach the runner set until someone
  changes that setting for the group. See GitHub's
  [runner group access](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/manage-access).
  The group must also allow every repository you want routed there.
* **Docker** needs Docker-in-Docker mode (see [Docker](#docker)).
* **Linux only.** Runner sets run Linux jobs, on the architecture of the nodes
  they are scheduled to.

## Related

* [AutoPilot for CI](https://docs.costgraph.ai/costgraph/autopilot/ci.md)
* [AutoPilot for CI settings](https://docs.costgraph.ai/costgraph/autopilot/ci-settings.md)
* [CostGraph operator](https://docs.costgraph.ai/costgraph/operator/overview.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.