Verify a record yourself

Every audit ends in a signed, hash-linked record. Below are twelve of them, from runs against libraries we did not write. You can check each one on your own machine, offline.

Do it

go install github.com/pdbethke/corralai/cmd/corral@latest
curl -O https://corralai.dev/records/record-19.json

corral certify verify record-19.json \
  --pubkey 909429443700441a9f662c07c5f1b7110333fd7fb94fb21e8991e40f2cb6dc09 \
  --allow-unanchored

No API key. No network after the download. Nothing is sent anywhere.

Use v0.3.4 or later. Every earlier release — including v0.3.3 — verified the statement embedded in the signature but never checked it against the readable copy in the file, so a hand-edited record still printed verified. We found that while preparing this page. If you already have an older corral, re-run the go install line above.

What each record is

record	library	commit	herd
18	vercel/ms	4ff48ce	Claude (planter + writer), Claude critic
19	vercel/ms	4ff48ce	Gemini, no critic
20	vercel/ms	4ff48ce	Gemini planter + writer, Claude critic
21–23	minitest/minitest	581e7d5	Gemini
24–26	debug-js/debug	f405ade	Gemini
27–29	wvanbergen/chunky_png	7a1faf6	Gemini

Record 20 is the interesting one: the model that planted the faults and the model that judged the suite are from different vendors. That is the claim this whole tool rests on, in a file you can check.

Each record is an in-toto statement naming the repository and the exact commit as its subject, the role→model assignment as builder dependencies, and a hash-linked step ledger. Change one byte of any of it and the signature fails.

The other kind: an attestation from CI

The record above is signed by a key on the machine that ran the audit. That is the right shape for a local run and the wrong one for CI, where the machine is disposable: a fresh key per run chains to nothing, and a signature nobody can tie to an origin proves only that something signed it.

A run of the GitHub Action with attest: "true" instead publishes the verdict through GitHub’s attestation API, signed keylessly by the workflow’s own OIDC identity. Verifying it answers which repository, which workflow, which commit — without trusting the log or the person who linked you to it:

gh attestation verify corral-audit-statement.json --repo <owner>/<repo>

gh attestation needs a recent GitHub CLI; it is absent in 2.45, which some distributions still ship, and an older CLI answers with a help dump rather than an error. The API needs no particular version:

DIGEST=$(sha256sum corral-audit-statement.json | cut -d" " -f1)
gh api repos/<owner>/<repo>/attestations/sha256:$DIGEST

Both are the same in-toto envelope; they differ in who vouches for it. The local record says this key signed this verdict. The attestation says this repository’s workflow produced this verdict at this commit, which is the claim a reviewer actually needs.

What this proves, and what it does not

It proves integrity. The verdict you are reading is byte-for-byte the verdict that was produced, and the numbers in it were not edited afterwards.

It does not prove independent authenticity. The signing key is ours, and we are also the ones publishing the public key above — so a reader who distrusts us entirely gains nothing from checking a signature against a key we handed them. That is a real limit and we would rather state it than let the word “signed” do work it hasn’t earned.

These records are signed but not publicly witnessed — that is why --allow-unanchored is required. Anchoring a signature in a public transparency log is what would close the gap above, because then the timestamp and the signature would be attested by someone who is not us. The code path exists; these particular records did not use it.

The one thing you don’t have to trust at all is the kill rate. It is not a model’s opinion about a suite — it is the exit code of that library’s own test command, run against each planted fault inside a sandbox. You can reproduce it: clone the repo at the commit named in the record, run corral against the same file, and compare. It will not match exactly, and that is expected — see below.

Expect the numbers to move

Faults are generated fresh on every run, so two audits of the same file with the same suite return different kill rates. Measured swing on one file: 0.55 to 0.80.

That is why there are three records for most repositories here rather than one. A single run is evidence of specific gaps, never a grade — and anyone quoting one of these numbers as a score for a library is misreading it.

The honest footnote on chunky_png

Records 27–29 required one change to the repository: spec_helper.rb calls bundler/setup, bundler was not installed on the machine that ran the audit, and that line was replaced with a $LOAD_PATH adjustment. The library and its specs are untouched, but it is not quite “third-party as shipped,” and you should know that before treating it as a clean-room result.