Agent-written pull requests arrive with their own verification already claimed: “147 tests green”, “verified on three platforms”, “all review findings addressed”. You cannot cheaply check any of it. The PR is large, the environment it claims to have tested on is one you do not have, and the author may be a burner account driving a swarm. Your real options are blind trust or hours of manual re-review, and neither scales with the rate these PRs now arrive.
harnessie verify is a single command that re-litigates those claims. It runs your deterministic checks first, sandboxed, with exit codes as the only accepted evidence. Then it hands the artifacts to a fresh-context verifier model that never sees the author’s narrative: the PR body is treated as a list of claims to test, not as evidence. The command returns an exit code that fails closed, so “I could not verify this” is never reported as a pass. CI validators and output guardrails check one model’s output against a schema; this checks a finished artifact against the claims made about it.
Quickstart
pip install harnessie
# workspace = the checked-out PR head
# claims.md = the PR body's claims, one per line
harnessie verify \
--workspace ./pr-checkout \
--criteria claims.md \
--check "python3 -m pytest tests/ -q" \
--models models.yaml
echo $? # 0 verified, 1 failed, 2 cannot verify
The report answers claim by claim: reproduced, refuted, or not verifiable in this environment, with the evidence named. Stage the PR’s diff into the workspace (git diff <base>..HEAD > PR.diff) and change-surface claims (“docs-only”, “additive”, “only touches these two files”) become checkable too.
The verifier model comes from a one-file YAML config and can be any OpenAI-compatible endpoint, including a local one. Untrusted diffs never have to leave your machine.
How the two layers divide the work
The deterministic layer runs the commands the PR itself names as proof: the test suite, the linter, the build. They execute inside an OS sandbox under harness control, network-denied unless you opt in, and their full output is preserved as proof artifacts. If a claimed check fails here, the verdict is already in and no model is consulted.
The model layer covers what exit codes cannot see. A test suite passing says nothing about whether the diff matches the description, whether “additive only” is true, or whether a security claim’s reasoning holds. The verifier role is limited to reading and test execution. It judges only artifacts against criteria, and it is required to end with a machine-parseable verdict. Anything else, including a missing verdict, fails closed. There is no retry ladder in this mode: a finished artifact’s failure is the answer, not a prompt to try again.
Three exit codes carry the contract. Zero means every check passed and every claim was reproduced. One means something failed or was refuted. Two means the infrastructure could not produce a verdict: a missing config, an unavailable sandbox, a provider error. Neither pass nor fail was earned. That third code matters: a verifier that converts “could not check” into either confidence or alarm is lying in one direction or the other.
Does it actually refuse?
A verifier that always passes is a rubber stamp, so here is the tool’s first real run. Pointed at its own author’s pull request to Ringer, an open-source agent harness, it refuted one of the PR’s claims: the description said only two existing files were edited, and the staged diff showed a third. The description got fixed on GitHub, the verification re-ran, and only then did it exit 0.
That sequence was deliberate. If the first public act of a claim-checking tool is not checking its own author’s claims, the tool has not earned its exit codes. The decision record behind the feature is public: AIDR-0006.
When to use it
You maintain a repository receiving large, agent-generated PRs whose claimed verification you cannot reproduce in review time. You run agent swarms and want an intake gate stricter than “the worker said done”. You want a verification step whose contract is an exit code, which means CI can consume it and no prompt can switch it off.
When not to use it
Small human-written PRs you can read in five minutes do not need it. It does not replace your test suite; it runs your test suite and then interrogates what the tests cannot see. Claims that need an environment you do not have (another operating system, live credentials, a specific device) are reported as unverifiable rather than silently passed, so it will not manufacture certainty where none is available.
Links
The next time a pull request tells you it verified itself, the interesting question is what happens when something that never read the author’s story checks.