Don't Let an Agent Grade Its Own Work
If you ask an agent to build something, it will tell you when it's done, which is often not the reassurance it sounds like. The agent that wrote the code is also the agent deciding whether the code is right, and it makes that decision based on how well it remembers your original request, a request that gets easily lost during a long agentic workflow.
A better prompt can't fix this. The agent needs an independent, auditable, deterministic judge that it can't manipulate. Nerine, a small scripting language purpose-built for testing web applications, can be that judge.
Why Nerine should be the judge
Nerine is independent by design. A script doesn't come from the code it tests, and nothing about Nerine requires the agent to write it. Someone else, even a second agent, authors the script, before building begins. An LLM that writes its own tests only proves it can meet its own standard, which is what we want to prevent.
Nerine is auditable because it's small enough to read whole. It has three commands. `modify` changes the request, `extract` pulls a value from the response, `compare` checks it against what's expected. There is no SDK to remember and no client library to understand. A script short enough to read is short enough to audit, and no agent shortcut can hide from someone who reads the whole script.
Nerine is deterministic too. `compare` checks the response against a fixed operator, equals, contains, matches a regex, and their negations. None of those is a judgment call. Run the same script against the same response and it gives the same verdict every time.
Building an experiment
To test the hypothesis, I wrote a full specification for a small task list API. It covers sessions, cookies, CSRF, JSON validation, and a fixed order in which failures get checked. Then I wrote six Nerine scripts that check a finished server against that spec. They cover login and logout, full CRUD on a task, wrong or missing CSRF tokens, malformed bodies, two logins staying independent of each other.
Then I wrote instructions for an agent to build against the spec, including two rules:
Do not edit, delete, rename, or add to anything in tests. If you believe a script is wrong, stop and say so instead of changing it.
Do not write code whose purpose is to satisfy a test rather than the specification.
The first rule locks the tests and the second closes the loophole that locking alone would leave open. Even with the tests untouched, the agent could still write code that narrowly satisfies them instead of the specification. Both come down to the same thing. "Make the check pass" and "make the check pass by fixing the code" look identical from the agent's own perspective when nothing stops it from editing the tests.
The rule against editing tests is only as good as the agent's willingness to follow it, so it isn't sufficient on its own. It holds because the scripts live in the repository and are short enough to read at a glance. Any change to tests/ would show up in the same diff a person already reads to check the agent's work. Breaking the rule wouldn't go unnoticed.
Running the experiment
I asked Claude to build the server following the spec. When the tests fail, Nerine prints one line for each failure and the agent reads the error, fixes the root cause, and runs again. This cycle repeats until the output goes silent.
Here's what that loop looks like in practice. A test script checks that creating a task returns status 201 and includes the required fields.
POST http://127.0.0.1:8080/api/tasks
modify type application/json
modify body {"title":"write blog post"}
compare status == 201
compare body contains "id"
compare body contains "done"
On the first run, the endpoint doesn't exist.
ERR comparison `compare status == 201` failed line=4 script=02-create-task.txt test="POST http://127.0.0.1:8080/api/tasks"
ERR comparison `compare body contains "id"` failed line=5 script=02-create-task.txt test="POST http://127.0.0.1:8080/api/tasks"
ERR comparison `compare body contains "done"` failed line=6 script=02-create-task.txt test="POST http://127.0.0.1:8080/api/tasks"
The agent creates a POST handler for /api/tasks. On the second run:
ERR comparison `compare body contains "id"` failed line=5 script=02-create-task.txt test="POST http://127.0.0.1:8080/api/tasks"
ERR comparison `compare body contains "done"` failed line=6 script=02-create-task.txt test="POST http://127.0.0.1:8080/api/tasks"
The handler returns 201 now, but the response body is incomplete. The agent adds the missing fields. On the third run:
$
Silent. The test passes. The agent moves on to the next script. Each test becomes a concrete, auditable statement of what the server must do.
Why it works
Independently written tests keep the agent from grading itself. Auditable tests catch attempts to cheat. Deterministic test results keep the verdict the same no matter who runs the tests. Drop any one, and the agent's own account of how well it met the spec sneaks back in.
Six Nerine scripts can't prove an API is production-ready, but they can prove it meets the specification. The specification itself might miss edge cases a real system would need. But the tests are locked and auditable. You can read exactly what the agent verified and what it didn't. An agent building against something independent, auditable, and deterministic has nowhere to hide.
The full specification, the agent instructions, and all six Nerine scripts from this experiment are available on GitHub.