Trust

How we can be wrong

A verdict that moves money has to be checkable and its mistakes have to be known. Here is every wrong verdict we have found, where a person decides instead of the machine, and what is not built yet.

Your code on a pilot

What we receive. From the request form: your name, work email, company, your role and an optional note, so we can reply. For the pilot itself: read access to one Python repository, the bug reports to fix and, if you write them, the hidden tests. Nothing else is needed.
Where your code runs. On hosts dedicated to judging, inside a sandbox with no network and strict limits, three runs per fix so a flaky test cannot decide the verdict. The code under test cannot reach the hidden tests or the results (see below).
What leaves the sandbox. Only the outcome and SHA-256 digests of the code, tests and evidence, written into the signed receipt. Receipts carry digests, not your source code or test contents.
How long we keep it, and deletion on request. The exact retention period is being finalised and will be written here. Until then, ask us at hello@resultbond.com before the pilot starts and we will agree it in writing.
Python only. We judge Python projects with pytest. A repository in another language cannot be judged yet, so a pilot does not start on it.

What a verdict proves, and what it does not

PASS proves that this exact code passed tests the agent never saw, in all three runs, and did not break the repository's existing tests. FAIL proves it did not. HOLD means we could not tell, and a person decides.
A verdict does not prove that the architecture or code quality is good, that there are no other bugs, that the code is secure, or that it performs well. It covers only the behaviour the tests check.
Who can check the signature: anyone, without asking us, in the browser at /verify or with the open-source verifier (next section).

Don't trust us: check

Every verdict is a signed receipt. It binds the job terms, the exact code that was judged, the evidence and the decision with SHA-256 digests, signed with Ed25519. Change one byte and verification fails.
Anyone can check it. /verify runs in your browser and uploads nothing; the same check runs in Python and JavaScript and they agree on shared test vectors (how).
Pin one key. Our offline root key rb:root:2026-10 (k4kS-tpmIN5OQFjdt8Q8AJiaPC_iOa6jfhU5IIz3AUE) signs the list of working keys at /.well-known/resultbond-keys.json. Compare the fingerprint through a channel other than this site.

Every wrong verdict we know of

Where	What happened	Why	What we changed
Benchmark, `shop/forge-exit` 10 of 570 runs	A wrong fix was judged PASS.	With ordinary pytest tests the code under test runs in the same process as the tests and can forge the whole report.	Black-box tests: the code runs in a separate worker with no access to the results. The same attack and two stronger ones now FAIL, 10 of 10 each. Pilots use black-box tests by default; the forgeable mode stays in the benchmark as a known false PASS.
Demo pilot, round 1 2 bugs, 4 runs	Both agents' fixes for `bucket` and `ichunked` were judged PASS; the maintainers' own test suites said FAIL.	Each fix broke behaviour our hand-written tests did not cover.	Pilots also run the repository's own tests as a regression check (`--repo-tests`). Round 2: the broken fixes FAIL with “broke existing tests”.
Demo pilot, round 2 1 of 36 runs	Agent A's `ichunked` fix was judged PASS; the maintainers' suite said FAIL.	The maintainers also changed an error message the bug report never asked for. The agent fixed what was asked.	Counted against us anyway. In a client pilot such a requirement is written into the task before any agent runs.
Internal audit, October 2026	In black-box mode, code under test could read the hidden test files.	The hidden tests were mounted into the sandbox.	Fixed in 0.10.2, before any client job: the tests reach the sandbox over a private channel and are erased after collection. A benchmark attack proves the code cannot see them.

A correct fix has never been judged FAIL: 0 times in 570 benchmark runs and 72 pilot runs. On this corpus that bounds the rate at roughly 0.5 % (the “rule of three”); a wider mix of bugs will find some. That is why the next section exists.

Where a person decides

HOLD instead of a guess. If the original code behaves inconsistently, the fix behaves differently between runs, the sandbox fails, or the hidden tests were not proven to test the bug, the verdict is INCONCLUSIVE and the job is held. Nobody is blamed by default.
An independent reviewer closes held jobs with a signed decision: pay, refund, or return both deposits.
A window to dispute. A receipt settles only after its challenge window closes; a disputed receipt is reviewed by a person.
Hidden tests are qualified before any agent runs. They must fail on the original code, and pass on a known-good fix, so tests that would reject a correct fix written differently are caught first. A client's engineer can sign off on them; receipts say whether a person did.

How hidden tests stay hidden

The agent receives the bug report and the public tests only. Hidden tests are stored with the evaluator and are never sent to the agent or its platform.
Inside the sandbox they arrive over a private file descriptor, are loaded by the test runner and erased before the candidate's code starts; the code under test runs in a separate process without network.
Receipts carry only digests of the tests, never their contents.
Who writes them: the client (we keep them secret), our team as a service, or, for the public index, regression tests taken from real upstream fixes. Every source goes through the same qualification.

What is not built yet

Python and pytest only. Other languages when pilots ask for them.
The sandbox shares the host kernel (bubblewrap, cgroups, no network). A kernel exploit could escape it, so judging runs on hosts dedicated to it; microVM isolation is next.
The escrow contract has not been externally audited and runs on a testnet only. Money moves on a real network only after an audit and legal review; until then pilots are shadow mode and payment is by invoice.
Small team. Security reports: see the key policy and reach us through the request form.