Star
Search corralai.dev

The GitHub Action

The Action is how corral becomes a gate rather than something you remember to run. It installs itself, audits only the files a pull request touched, and writes the verdict to the run page verbatim.

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- uses: pdbethke/corralai@v1.0.0-rc.29
  with:
    test-command:  "go test ./..."
    gemini-key:    ${{ secrets.GEMINI_API_KEY }}
    mutant-model:  gemini-3.6-flash
    writer-model:  gemini-3.6-flash
    derive-model:  gemini-3.6-flash
    critic-model:  "off"

That is the whole workflow: one key, one model in every seat, the advisory critic off. corral has no default models, so the three seats are named here, not assumed. There is no corral for you to install — the action’s first step go installs it into a private GOBIN and puts it on PATH. The one requirement is a go binary already on the runner; GitHub-hosted runners ship one.

Pin a tag, not a branch

v0.3.0 and later carry an action.yml; v0.1.0 and v0.2.0 predate the action and do not. Prefer pdbethke/corralai@v1.0.0-rc.29 over @main, so a push to main cannot change what runs in your CI, or pin the commit SHA you reviewed if you want a reference a re-tag also cannot move.

fetch-depth: 0 is not optional

The changed-file set is computed with a three-dot range against the merge base, because that is what “what this PR changed” means. GitHub’s default checkout is depth 1 and has no merge base to find. On a shallow checkout the diff computation fails closed — exit 1, never a silent full-repo scan. A missing fetch-depth: 0 is the single most common way a first run breaks.

What it said about this repository

We ran it against the file that starts our own program, cmd/corral/main.go, as a gated check on a real commit:

kill rate 0.25 (30 survivor(s), 1 proven missed)
the pool's authored test PROVED 1 of 30 survivor(s) catchable by execution

Forty faults planted, ten caught, thirty missed, and one of those thirty proven catchable by a test the pool wrote itself. The run took 11m12s on a GitHub-hosted ubuntu-latest runner.

That number is bad and it is ours. A gate whose author can quietly not-publish the result is furniture.

What one run costs

Roughly (mutants × your suite’s whole runtime) per audited file. It scales with how long your tests take and how many files the PR touched — not with the size of the change. The 11m12s above is a measurement from one repo, not a law; your first run is the only timing that really applies to yours.

An earlier version of our own documentation predicted around two hours for that file. It was wrong by roughly ten times, in the direction that would have talked you out of trying it. We are leaving the correction visible because it is the same failure mode this tool exists to catch: a plausible number nobody executed.

Four levers keep a run bounded:

  • top — audit at most this many of the highest-ranked candidate files. On a pull request the diff narrows the candidates and nothing else bounds them: corral’s default of 25 applies only to a whole-repository run, so with top empty every changed file that pairs with a test is audited. Set it, and the changed files it cuts are listed as not audited.
  • path — audit only one directory of the repository, such as one crate of a Cargo workspace or one package of a monorepo. The checkout stays whole, so tests elsewhere still pair and the build still sees the root’s files.
  • paths: on the workflow — so a docs-only PR doesn’t spend time printing NOTHING IN SCOPE.
  • diff-base declares an empty default, and the action synthesizes a base only when GITHUB_BASE_REF is set — that is, on a pull request. On a workflow_dispatch or a push run there is no base to infer, so no --diff-base is passed and the whole repository is audited. On a pull request leave it alone; anywhere else, set it explicitly or expect a full scan.

A diff that touches no auditable candidate at all is a legitimate pass: the action prints NOTHING IN SCOPE: and exits 0. One exception, and it is deliberate: if the diff changed source files that corral could not pair with any test, or deleted the test that paired one, the run exits 1 rather than 0 — “nothing could be audited” because nothing was reachable is a result, not a pass.

Giving it teeth

By default a graded file exits 0 no matter what kill rate it measured — a file where every mutant survived merges as cleanly as a perfect one. That is deliberate: adding a default would silently change the exit code of every existing caller.

- uses: pdbethke/corralai@v1.0.0-rc.29
  with:
    test-command:  "go test ./..."
    gemini-key:    ${{ secrets.GEMINI_API_KEY }}
    mutant-model:  gemini-3.6-flash
    writer-model:  gemini-3.6-flash
    derive-model:  gemini-3.6-flash
    critic-model:  "off"
    min-kill-rate: "0.7"

The check is per file, not on the aggregate — a well-tested file elsewhere in the PR cannot average out or mask a weak one. 0.7 means at least 70%: a file at exactly 0.70 passes, 0.69 fails the run, and the report names every breaching file on its own line:

  KILL-RATE BREACH: 1 file(s) below --min-kill-rate 0.70:
    0.40  pkg/widget.go (0.30 below threshold)

Reach for this only once you have real timings from your own repo. A required check that can take hours is not a merge gate anyone will keep.

The review, on the pull request

Name reviewer-model and the same step also runs corral review on the change: a model that has never seen the repository is handed the changed directories and told to assume the code is wrong; every claim comes with a script the step runs, a claim that does not reproduce is demoted on the record, a verifier-model tries to refute what stands, and the review is one signed entry in the same ledger branch as the audit. review-fail-on: reproduced (the default) fails the check with exit 3 when a reproduced finding stands — after the entry is written. A seat can be an API model or a coding-agent CLI (claude-code, codex) if you install the CLI on the runner and pass its login through claude-code-oauth-token or codex-api-key; neither has been run on a runner yet. The full reference has the inputs and what “the change” means.

Where the verdict shows up

The report goes to the job summary — the page you land on when you click the check — as the report verbatim, not a rendering of it. The kill rate, the weakest files, and the lines that qualify what those numbers mean (NOT AUDITED, DID NOT FINISH, WRITER FAILED, TEST UNSOUND) are the same bytes corral printed. A second renderer would be free to drift, and drift in a summary always flatters the run, because the lines that get dropped are the qualifying ones.

It uses $GITHUB_STEP_SUMMARY, so it needs no permissions: block and works on fork pull requests where a PR-comment token doesn’t exist. The report reaches the summary even when the run fails — a red X whose reason was discarded is the problem this exists to fix.

Who pays, and how not to pay for a stranger

The bill lands on the repository that runs the workflow: your runner minutes, your API key. On a public repo an outside contributor’s pull request would spend your money, once per push, for as long as they keep pushing.

GitHub withholds secrets from fork pull requests, so an audit on a fork has no key and goes red (needs a Google key, exit 2) rather than skipping — a contributor sees a failed check that is your configuration, not their code. Guard the step with an if: on the head repository, as below, so a fork PR is skipped on purpose. And don’t rely on the missing secret alone: the specific way it gets undone is someone reaching for pull_request_target because “fork PRs skip.” That trigger runs with your secrets while checking out the contributor’s code, so it does not merely spend your key, it exposes it. If fork PRs skipping looks like a bug, it is the feature.

Say it in the workflow instead, and add an opt-in so no pull request — yours included — starts a paid job merely by existing:

on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]
    paths: ["**.go"]
  workflow_dispatch:

jobs:
  audit:
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
       contains(github.event.pull_request.labels.*.name, 'audit'))

.github/workflows/self-audit.yml in this repository is that shape — non-blocking, top: "1", Go-only paths, fork-guarded and label-gated — plus a step that checks the key secret is present and skips the audit with a notice when it is not.

Keys are additive

anthropic-key, gemini-key and openai-key are not alternatives; set as many as your role routing needs. An unset key is never exported as an empty variable, and no key value is ever echoed.

A key alone does not move providers. corral routes each role to its own model and has no default models, so a key with no seats named refuses to run, naming the credentials it can see. Set derive-model, writer-model, mutant-model and critic-model (a model, or off). The critic must differ from the writer — that decorrelation is enforced, not advisory — and critic-model: off disables it entirely, which is reasonable when one vendor gives you only one usable model. The critic never gates the verdict either way.

A receipt a reviewer can verify

A job summary is a screenshot. attest: "true" publishes the verdict as a signed in-toto statement through GitHub’s attestation API, and attaches the statement itself to the run as an artifact:

permissions:
  contents: read
  id-token: write
  attestations: write

# ...
    attest: "true"

Signing is keyless, through the workflow’s own OIDC identity. That matters more than the format. A certify --local run signs with a key generated on the machine, so on a disposable runner every audit is signed by a fresh key that chains to nothing; an attestation chains to the repository, the workflow and the commit — which is what a reviewer actually wants to check. Free on public repositories.

Anyone can then verify it without trusting the log, or you:

gh attestation verify corral-audit-statement.json --repo <owner>/<repo>

gh attestation needs a recent GitHub CLI — it does not exist in 2.45, which some distributions still ship, and an older CLI answers with a help dump rather than an error. The same answer straight from the API, with no CLI version to care about:

DIGEST=$(sha256sum corral-audit-statement.json | cut -d" " -f1)
gh api repos/<owner>/<repo>/attestations/sha256:$DIGEST

What the statement says

Every audited file’s kill rate, survivors and proven gaps — with the flags that say what a zero means (timedOut, testWriterFailed, poolTestUnsound, because provenMissed: 0 means nothing was proven rather than the suite is clean whenever one is set) — plus the thresholds it was judged against, the model in each role, and the audited-of-candidates denominator, so a clean result cannot be flattered by omitting how little was looked at.

It is written before the gate’s exit code is honoured, and the attest steps run under always(). A receipt you only keep when the verdict flatters you is not evidence, and the failing runs are the ones a reviewer most needs.

Sending verdicts to your own warehouse

One audit answers is this change defended. It cannot answer whether your tests are doing anything across your projects over time, and that needs somewhere to put the rows:

    push: "md:my_database"                     # or a DuckDB path
    motherduck-token: ${{ secrets.MOTHERDUCK_TOKEN }}

The warehouse is yours. corral has no hosted tier and collects nothing — your key, your runner, your database — and any DuckDB works, so this is a destination rather than a lock-in.

It is append-only, because a receipt you can UPDATE is not a receipt and overwriting is how a trend is lost. Every row carries the run URL, and — only when attest: is also set — the sha256 of the signed statement it came from, so a row traces back to something a third party can verify. Without attest:, statement_sha256 is left empty rather than fabricated.

What it is for is the question a single run cannot answer. A kill rate is a sample: mutants are generated fresh each time, and the same unchanged diff has scored 0.85 and then 0.90 here. Forty rows are a distribution — “this file drifted from 0.9 to 0.6 over two months” is a claim no individual run supports. See the warehouse for the queries worth running.

The gate that does not flap

min-kill-rate is the obvious merge gate and the wrong default. A kill rate is a proportion of freshly generated mutants, so it moves between runs on code nobody touched — the identical diff of one of this project’s demonstration pull requests scored 0.85 and then 0.90, the higher number on the more weakened suite (CHANGELOG, --max-proven-missed). Set the threshold near a healthy value and it goes red on good work; the team turns it off, and you have no gate at all.

max-proven-missed keys on the claim that does not move:

    max-proven-missed: "0"

A proven-missed gap is a survivor the herd then killed with a test it wrote and ran. Not a proportion, not an opinion — a specific planted fault your suite does not catch, with a test shown by execution to catch it. It is a gap in the tests, not a bug found in your code, and a person still decides whether the test belongs. 0 means any demonstrated gap fails the build.

It fails closed. If the herd had survivors but could author no test that graded them, proven_missed reads 0 because nothing was proven — not because the suite is clean. The run reports that case separately (PROVEN-GAP UNMEASURED) and fails rather than passing on a question nobody answered.

Both gates compose: min-kill-rate as a loose floor, max-proven-missed as the thing that actually blocks a merge.

One key, if that is all you have

Three provider inputs and a decorrelation rule read like corral needs an account with every vendor. It does not. The gate’s verdict is measured by execution — the critic is advisory and never gates it — so a single key runs the whole thing:

- uses: pdbethke/corralai@v1.0.0-rc.29
  with:
    test-command:  "pytest -q"
    gemini-key:    ${{ secrets.GEMINI_API_KEY }}
    mutant-model:  gemini-3.6-flash
    writer-model:  gemini-3.6-flash
    derive-model:  gemini-3.6-flash
    critic-model:  "off"
    shadow-model:  "off"
    min-kill-rate: "0.8"

That is a real gate. Faults get planted, your suite runs against every one of them in a private copy of the checkout on the runner (the Action uses the workspace substrate, not the bwrap jail — a runner is already disposable), and the kill-rate either clears the bar or fails the check.

What you give up, and corral says so out loud. With every graded seat on one vendor there is no independent read, and the verdict block prints:

decorrelation: every graded seat is google — if this code was WRITTEN by a
google model, the same lineage planted the faults and graded the tests. Point a
role at another vendor (--critic-model / --mutant-model) for an independent
read.

That warning is the point of the mode, not an apology for it. If an agent from one lineage wrote the change, having the same lineage decide what counts as a fault is a narrower check than it looks — the failures a model cannot imagine in its own output are exactly the ones it will not plant. A second vendor in any one seat costs one more key and buys back the independence.

A disabled critic reports as disabled. The verdict says critic review: not run — no test-critic was assigned, never “no vacuous tests flagged”. An absent reviewer does not get to hand out a clean bill of health.

Exit codes

  • 0 — the scan graded at least one file and every audited file met min-kill-rate and max-proven-missed (when given); or nothing in scope. With both gates unset, a weak-but-gradable suite still exits 0 — read the report for the number.
  • 1 — a real failure: the diff changed source files corral could not pair with a test (NOT AUDITED:); files were in scope and none could be graded (COULD-NOT-GRADE:, e.g. every candidate’s baseline suite was already broken or flaky); every audited file timed out; enumeration failed; a file scored below min-kill-rate (KILL-RATE BREACH:); or a file broke max-proven-missed, including PROVEN-GAP UNMEASURED.
  • 2 — the run never started: a usage error, an unnamed seat, a seat with no key for its provider (what a fork’s pull request gets), or a test-command containing one of corral’s own flags.
  • 3 — review-fail-on: reproduced and a reproduced review finding stands.
  • 130 — the run was interrupted; the audit records nothing.

The full reference

Every input, the quoting rules for test-command, why the action deliberately avoids actions/setup-go, and why files are graded one at a time on this substrate are documented in docs/corral/github-action.md in the repository.