Star
Search corralai.dev

Your first audit, in detail

This walks one real corral certify --local run start to finish, using the same password-validator example published in the field note “Terribly sorry to trouble you… but your tests, um, well. They suck, really”. The code, the test and the command are the real ones; the verdict lines further down are in the format the run prints, with illustrative numbers — your run draws its own mutants and will score differently.

The code under review

// SPDX-License-Identifier: Elastic-2.0
package passwd

import "unicode"

// Valid reports whether p is a valid password: length >= 12 AND it contains an
// uppercase letter, a lowercase letter, a digit, and a symbol.
func Valid(p string) bool {
	if len(p) < 12 {
		return false
	}
	var up, lo, di, sy bool
	for _, r := range p {
		switch {
		case unicode.IsUpper(r):
			up = true
		case unicode.IsLower(r):
			lo = true
		case unicode.IsDigit(r):
			di = true
		case unicode.IsPunct(r) || unicode.IsSymbol(r):
			sy = true
		}
	}
	return up && lo && di && sy
}

The developer’s own test

A test that only ever feeds a valid password:

func TestGappy_ValidLength(t *testing.T) {
	if !Valid("Abcdefgh1!xy") {
		t.Fatal("a 12-char password was rejected")
	}
}
func TestGappy_TooShort(t *testing.T) {
	if Valid("Ab1!xyz") {
		t.Fatal("a 7-char password was accepted")
	}
}

Green in CI. Looks reasonable — it checks a boundary (7 chars vs 12) and a happy-path accept.

Running it

export ANTHROPIC_API_KEY=sk-ant-...

corral certify --local \
  --code passwd.go \
  --test passwd_gappy_test.go \
  --writer-model claude-sonnet-5 \
  --mutant-model claude-sonnet-5 \
  --critic-model claude-haiku-4-5 \
  --goal "Valid must require length >= 12, an uppercase letter, a lowercase letter, a digit, and a symbol" \
  --out verdict.json \
  -- go test ./...

The model names above are an example, not a default — corral has none. Every seat is yours to name, from whichever provider you hold a key for; the only rule is that the test-critic must differ from the test-writer.

No API key? Any seat can be a coding-agent CLI on its own subscription login instead: --writer-model codex --mutant-model codex --critic-model claude-code (see Getting started).

--code/--test may be absolute or relative paths. On Ubuntu 24.04 and later, if bwrap won’t start (AppArmor restricts unprivileged user namespaces by default), load the profile Ubuntu ships for it, as the error message and Getting started describe — or add --jail container and export CORRALAI_EXEC_IMAGE=golang:1.26 to run the jailed check in Docker instead, with no sudo required.

What happens inside the jail

  1. Preflight. The Go toolchain is checked (Preflight() fails closed if go isn’t on the jail-visible PATH), and the sandbox backend is resolved — bwrap by default on Linux, refusing to fall back to unsandboxed if it can’t start.
  2. Mutants. A mutant-generator model seeds goal-violating mutants into Valid — dropping the digit check, weakening the length bound, flipping the symbol test, and so on. On a file with several functions the generation fans out: one seat per group of functions (balanced by complexity), so every function is probed rather than just the one a single generator picked — --max-shards bounds the width, and defaults to 8 (only --swarm sizes itself to your machine). A mutant that makes the suite hang (a broken loop bound, say) is killed fast and counted as caught, so a non-terminating mutant can’t stall the run.
  3. Dev kill-rate. The existing test above is run, in the jail, against every mutant. It only ever calls Valid with a password that already satisfies every rule (or is simply too short) — so a mutant that loosens a rule (say, dropping the digit requirement) still passes: the input never exercised the rule it broke. Several mutants survive — a survivor is a planted mutant your suite didn’t catch. corral discloses it unadjudicated: it may be a real untested edge, or an equivalent mutant (a change nothing can observe, so no test could ever catch it). Which one is the human’s call, not the tool’s — corral won’t call an unproven survivor a real bug.
  4. Test-writer. For each surviving mutant, a test-writer model (a different model than the critic, enforced — decorrelation isn’t optional) attempts a targeted test; if its first attempt doesn’t compile, the compiler’s own error is fed back so it corrects rather than blindly repeating. When the jail confirms the test kills the mutant, that survivor is now a proven, catchable gap — not a fluke of the harness — and the test is handed back for you to add.
  5. Test-critic. A decorrelated critic model reads the dev’s test file next to the code and returns a verdict per test (sound, vacuous, or a dead check); for Go, Python, PHP and Rust it is handed every test name and must judge each one. It flags what it would have said in review: the suite never tries an invalid password that’s the right length. Its findings are carried as an unverified second opinion — advisory, never gating the signed verdict, which is execution-proven only.
  6. Shadow challenger (only if you name a --shadow-model, or a --shadow-pool of candidates from which one is drawn per run and disclosed in the record). In parallel, a second model attacks the same regions — a same-ground, head-to-head measurement of one model against another. It’s recorded for comparison and never enters the verdict; the exam’s difficulty is set by the primary model alone, so certification means exactly what it did before.

Once the mutants exist, the run converges to a signed verdict — even if the herd can’t author a compiling test to kill a survivor, or the writer’s model call fails outright, it routes to needs-review and hands back what it found rather than spinning. The one seat that cannot be absorbed is the mutant-generator: if its provider fails there is no exam to grade, and the run exits 1 with the provider’s error and no verdict.

The verdict

  status:        NEEDS-REVIEW (dev suite killed 6/8 mutants)
  dev_kill_rate: 0.75
  95% interval:  0.41–0.93 (n=8 distinct)
  survivors:     2

The certify bar is a kill rate of 0.8. This one is below it, so the run routes to a human instead of auto-certifying — the honest failure mode. The kill-rate is measured, not asserted: it’s mutation testing executed in the sandbox, never the model’s self-report of how good the test is.

The interval is computed over the distinct edits that were graded (n=8 distinct), and it describes this sample of generated faults, assuming independent draws; it is not a calibrated probability that the software is correct. Generated edits can share blind spots and are not necessarily independent or representative. See the measurement limits. Review any generated tests and run them in your full suite before merging; passing individually does not guarantee that they work together.

The signed record

The verdict is written to a local, tamper-evident ledger (the same certify chain corral certify uses for the non---local path) and printed to stdout — subject (repo/commit), the models that played each role, the kill-rate, and the status. Because you passed --out, the signed record is also written as a self-contained file, which re-verifies any time, fully offline (the run prints this line for you, with your key filled in):

corral certify verify verdict.json --pubkey "$(corral certify pubkey)" --allow-unanchored

--allow-unanchored is required: a --local record is signed by your own key but never submitted to a public transparency log, so verify makes you opt in to that weaker “signed-by-you, not third-party-witnessed” trust explicitly.

Try it on your own code

Point --code and --test at a real file and its test, describe the guarantee in --goal, and pass whatever check command actually exercises it (go test ./..., .venv/bin/python -m pytest, npm test, cargo test, …). For anything with imports, add --repo-dir .. See Getting started for the jail-visibility gotcha (system-installed toolchain, not a --user/snap install), the in-project virtualenv for Python, the Rust rules and the Ubuntu AppArmor fix if bwrap won’t start.

Reproduce a published audit yourself

The sportspicker-core tapes on the recordings page are the easiest published runs to reproduce, because the subject is a public, zero-dependency library and the audit needs nothing but pytest. It scores sports pick’em contests, and the guarantee it has to defend is one anyone can hold in their head: a contest contributes exactly its budget — whatever scoring rule is used, however many people played, the points awarded in a round sum to that round’s pot.

git clone https://github.com/pdbethke/sportspicker-core && cd sportspicker-core
python3 -m venv .venv && .venv/bin/pip install pytest

corral certify --local --repo-dir . \
  --code sportspicker_core/awards.py --test tests_core/test_awards.py \
  --goal "A contest contributes exactly its budget: for any round, the points awarded across the field sum to that round's pot — whatever normalization strategy is chosen, however many members played, and however they scored, including a round nobody scored in. min_participants gates on members who SCORED, not members who merely submitted picks." \
  --mutant-model gemini-3.6-flash --writer-model gemini-3.6-flash \
  --critic-model claude-haiku-4-5-20251001 \
  -- .venv/bin/python -m pytest tests_core -q

If you want to see what the library actually does before auditing it, there is a command line over the same arithmetic in sportspicker-cli — one dependency, which is the library itself:

sportspicker award round.json --budget 1000 --rounds 4
#   pot 250.0000   awarded 250.0000

Change the strategy and the split changes while that total does not. That is the guarantee the audit is grading, visible in one command.

main scored 18/20 — CERTIFIED. Then run the identical command on git checkout demo/bug-counts-pickers, a branch that carries a real bug and a green 80-test suite: 15/20 — NEEDS-REVIEW. Same file, same command, and CI is happy with both. That gap is the whole product in one command.

Your numbers will not necessarily match ours — the mutants are generated fresh each run, so these recorded verdicts are not constants. Even the direction of a fresh comparison can change; a thinner suite is not guaranteed to score worse on a different generated exam. Replay the same recorded mutants to compare suites.