Run CI jobs on your own cluster - CostGraph Documentation

Preview. Runner sets on your own cluster are not routed yet. This page describes how they work after the feature ships, so you can prepare a cluster ahead of time.

AutoPilot for CI can send a job to a runner set in your own Kubernetes cluster, next to GitHub-hosted runners and runner providers. A runner set is a group of self-hosted runners that GitHub’s actions-runner-controller creates. It starts one short-lived runner pod per job, scales between its minimum and maximum number of runners, and removes each runner when its job finishes. CostGraph does not run or change the runners; it reads them and decides when a job goes there.

What CostGraph does

Finds your runner sets. The CostGraph operator already running in the cluster reports every runner set, its name, the GitHub organization or repository it serves, the CPU and memory each runner requests, and whether Docker is available. No extra setup in CostGraph.
Offers them as runners when routing. A runner set’s name is the runner label AutoPilot returns. A running runner set is a candidate for every job of the organization or repository it is registered for.
Prices each job at the node’s share. A job costs the node’s hourly price times the larger of its share of the node’s CPU and its share of the node’s memory. When a node has enough unused capacity for the runner, that capacity is already paid for, so the job counts as $0 for routing and savings. The node’s share is still recorded and shown next to every run, so you can see what the cluster spent.

Each routed job’s cost comes from the node its runner pod actually ran on.

Create a GitHub App for actions-runner-controller

actions-runner-controller needs its own GitHub credential to register runners. It must be one you own: CostGraph’s GitHub App isn’t used, because its private key would have to live in your cluster. We recommend a GitHub App over a personal access token. A runner set registered for a GitHub enterprise can’t use a GitHub App; use a personal access token for it. GitHub’s authentication guide has the details.

Create the app

In GitHub, go to your organization’s Settings > Developer settings > GitHub Apps and click New GitHub App. Clear Active under Webhook; the app needs no webhook.

Set its permissions

For runner sets registered for the organization, set Organization permissions > Self-hosted runners to Read and write and Repository permissions > Metadata to Read-only. For runner sets registered for one repository, set Repository permissions > Administration to Read and write and Metadata to Read-only instead.

Install it on the organization

On the app’s page, click Install App and install it on your organization, for all repositories or the ones you select.

Note the IDs and generate a private key

The App ID is on the app’s General page. The installation ID is the number at the end of the installation’s URL, https://github.com/organizations/<org>/settings/installations/<installation ID>. On the General page, click Generate a private key and save the .pem file.

Create the Kubernetes secret

Create the secret in the namespace the runner set is installed in. With the CostGraph operator chart, that’s the release namespace, costgraph. If you install actions-runner-controller yourself, use your runner set’s namespace instead.

kubectl create secret generic arc-github-app \
  --namespace costgraph \
  --from-literal=github_app_id=123456 \
  --from-literal=github_app_installation_id=654321 \
  --from-file=github_app_private_key=path/to/private-key.pem

Set githubConfigSecret: arc-github-app in the runner set’s values, as the operator chart example in Install actions-runner-controller shows.

Or use a personal access token

Create a personal access token (classic) with the admin:org scope for an organization runner set, or the repo scope for a repository runner set. Store it in the same secret as github_token:

kubectl create secret generic arc-github-app \
  --namespace costgraph \
  --from-literal=github_token=YOUR_TOKEN

Install actions-runner-controller

Use either of these.

Install it yourself
With the CostGraph operator chart

Follow GitHub’s quickstart and deploy a runner scale set. Install it before or after the CostGraph operator; the operator finds the runner sets either way.

Set arc.enabled: true in the operator chart’s values to install actions-runner-controller 0.15.1 and one runner set with the same release. The arcController and arcScaleSet values are passed unchanged to GitHub’s controller and runner scale set charts, so every option in GitHub’s documentation applies.The chart names the controller’s service account costgraph-arc-controller and expects the release in the costgraph namespace. If you install the operator in another namespace, set arcScaleSet.controllerServiceAccount.namespace to it.

arc:
  enabled: true
arcScaleSet:
  githubConfigUrl: https://github.com/acme
  githubConfigSecret: arc-github-app
  runnerScaleSetName: acme-k8s-4cpu
  minRunners: 0
  maxRunners: 10
  template:
    spec:
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner:latest
          command: ["/home/runner/run.sh"]
          resources:
            requests:
              cpu: "4"
              memory: 16Gi

arc-github-app is the secret from Create a GitHub App. The runner set runs in Docker-in-Docker mode by default (containerMode.type: dind). For more than one runner set, install GitHub’s runner scale set chart once per extra set.Helm installs the controller’s custom resource definitions only on a fresh install. To turn on arc.enabled in an existing release, apply them before helm upgrade:

helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --version 0.15.1 --untar
kubectl apply --server-side -f gha-runner-scale-set-controller/crds

Before you uninstall the release or turn arc.enabled off, delete the runner set first, so the controller can remove its runners from GitHub:

kubectl delete autoscalingrunnerset --all --namespace costgraph

Helm leaves the actions-runner-controller resource definitions in the cluster. To remove them after you uninstall, run:

kubectl delete crd autoscalingrunnersets.actions.github.com \
  autoscalinglisteners.actions.github.com \
  ephemeralrunnersets.actions.github.com \
  ephemeralrunners.actions.github.com

Set CPU and memory requests on the runner container. AutoPilot matches jobs on them; without requests it uses what the job asks for.

Docker

Jobs that need Docker go only to runner sets in Docker-in-Docker mode (containerMode.type: dind, the operator chart’s default). Runner sets without it, including the kubernetes container mode, get no Docker jobs. See GitHub’s runner scale set guide for the modes.

How routing picks it

A runner set competes with every other runner under the same rules:

The workflow’s policy (cheapest, fastest or balanced) applies unchanged.
The first time a job could run there, it runs once as a trial. The trial fails when it costs more than the default runner, or takes longer than the default runner’s time plus the larger of 5 seconds and 10%. A failed trial is skipped for that job until its price is lower than the price it had at the trial.
Settings and provider limits apply, with all runner sets under the kubernetes provider. Set allow_secrets: false there to keep jobs that use secrets off your cluster.
If no runner picks a routed job up in time, for example because the runner set is at its maximum, AutoPilot moves the run back to its default runners (see Stuck jobs).

A runner set that is not running is skipped with the reason “runner set on your cluster is not running”.

Limits

Enterprise runner sets. A runner set registered for a GitHub enterprise can be routed to for every organization of that enterprise you connect to CostGraph.
Public repositories. Runner sets join the Default runner group unless you set another. The Default group does not allow public repositories, so jobs from a public repository don’t reach the runner set until someone changes that setting for the group. See GitHub’s runner group access. The group must also allow every repository you want routed there.
Docker needs Docker-in-Docker mode (see Docker).
Linux only. Runner sets run Linux jobs, on the architecture of the nodes they are scheduled to.

​What CostGraph does

​Create a GitHub App for actions-runner-controller

​Or use a personal access token

​Install actions-runner-controller

​Docker

​How routing picks it

​Limits

​Related

What CostGraph does

Create a GitHub App for actions-runner-controller

Or use a personal access token

Install actions-runner-controller

Docker

How routing picks it

Limits

Related