How to evaluate a self-improving AI agent
A rising score can mean a better agent, more spending, or a broken test. Design the evaluation so those explanations can be told apart.
Updated
Write the evaluation contract first
A self-improving agent needs an evaluator that stays outside the part it can change. Define success before the first proposal: which tasks count, which artifacts prove completion, how much time and compute are allowed, and which regressions cause rejection.
For a coding agent, success might mean an independently executed test suite passes and a required artifact exists. For a research agent, it could mean a result is reproduced from recorded inputs. A fluent report that says the job succeeded is not an execution receipt.
We would start with a small set of real failure cases and known-good tasks. Make the first evaluator simple enough to audit by hand. An elaborate grading system can hide a bug just as easily as an elaborate agent.
- Candidate
- A bounded patch
- Independent checks
- Correctness, tasks, cost
- Decision
- Promote or reject
Separate search, selection, and final evaluation
Use development tasks to produce candidates. Use a separate selection set to choose promising versions. Keep a final set untouched until you are ready to assess the selected version. The distinction is about how the data is used, not what the folder is called.
If final-set scores influence another round of edits, that set is now selection data. Record the change and create fresh final tasks for the next claim. Otherwise a long search can gradually memorize the test while the report still calls it unseen.
Balance the task mix around the intended user. A win on short Python edits may not transfer to long GPU jobs, missing dependencies, or recovery from a machine failure. Report scores by task family as well as in aggregate.
Compare at equal cost and equal opportunity
Choose a resource budget before comparing versions: total tokens, wall time, tool calls, or measured cost. Often you need more than one view. Quality at a fixed dollar budget answers a different question from quality at a fixed latency.
Charge retries, failed tests, and setup to the candidate that used them. Also report the one-time cost of finding the new agent separately from its per-task cost. An agent can be cheaper to use and still expensive to discover.
For stochastic tasks, run more than one seed or repeat. Do not call a small difference decisive without checking variation. If a score is near a promotion threshold, collect more evidence or keep the prior version.
Protect the evidence path from the candidate
Run the grader from a separate, read-only version of its code. Keep held-out inputs and final result writes outside the candidate's permissions. Save raw tool exits and outputs through the runner, rather than asking the model to restate what happened.
The Darwin Gödel Machine report includes examples of reward hacking. That is a concrete reason to test your evaluator's failure behavior. Give it a candidate that returns fake logs, deletes an expected file, emits a malformed score, or exits early. Each should fail for the reason you expect.
Sandboxing is one layer. You also need an independent budget stop and a clean distinction between “the test failed” and “the system could not run the test.” Missing evidence should not become a zero-cost success.
Test the evaluator before running the agent
We built a small, downloadable Python example to make the evidence boundary concrete. It runs 13 fixed programs against six integer-addition tasks, captures their actual process outputs, and grades the answers independently. It uses Python 3.10 or newer and the standard library, with no model calls, network requests, GPU, or package installation.
Download and inspect the script, save it as evaluator-smoke-test.py, then run the command below. It creates a new directory containing report.json, the task inputs, expected answers, each fixture's source, and raw stdout and stderr. An existing output directory is rejected so a rerun cannot overwrite the first record. Choose a different directory name for each run.
The deliberately bad programs try different ways of looking successful: printing a fake test log, reporting their own score, returning only one answer, or producing the right answers before exiting with an error. The grader accepts only a complete JSON object containing integer predictions. It rejects duplicate keys, non-finite numbers, and booleans; Python's default JSON decoder permits some of these values, so parsing alone is insufficient. The runner owns exit and timeout status, and the grader computes the score from the full task set.
python3 evaluator-smoke-test.py --out evaluator-run
python3 -m json.tool evaluator-run/report.jsonWhat the local test actually caught
Our September 11, 2026 local CPU run matched all 13 expected outcomes. Only the correct program was accepted. The program returning six zeroes received 1/6 accuracy and was rejected; invalid, missing, or interrupted results received no score. The remaining cases are protocol failures, not zero-accuracy measurements. The downloadable report includes the source and input hashes, environment, process exits, elapsed times, and expected versus observed outcomes.
The script's final 13/13 means that the evaluator behaved as expected for these fixtures. It does not mean 13 agents passed. This is a synthetic smoke test of the result protocol, not an AI benchmark or evidence of recursive improvement. The fixtures are public, so they cannot support a held-out generalization claim.
| Fixture | Observed result | Why it matters |
|---|---|---|
| Correct answers | Accepted; score 1 | Known-good control establishes the success path |
| Six zeroes | Incorrect; score 1/6 | A valid output can still fail the task |
| Fake log or malformed JSON | Invalid JSON; no score | A claim of success is not task evidence |
| Self-reported score, partial answers, boolean answer | Invalid schema; no score | The grader owns the score and expected task count |
| NaN or duplicate prediction key | Invalid JSON; no score | Ambiguous or non-finite values fail explicitly |
| No output or oversized output | Missing output or output too large; no score | The result must satisfy the output contract |
| Correct output followed by exit 1 | Nonzero exit; no score | Output cannot override the runner's failure status |
| Program exceeds one-second deadline | Timeout; no score | An unfinished attempt cannot become a success |
Where this example stops
We intentionally run only the fixed, trusted fixture programs bundled in the file. A Python subprocess is not a security sandbox. The one-second timeout applies to these simple child processes; the output-size check happens after capture and is not a memory limit. Do not plug arbitrary generated code into this local runner and assume the grader, files, network, or process tree are protected.
For a real agent loop, put candidate execution behind an independently enforced boundary: separate permissions for evaluator code and final evidence, restricted data access, resource limits, and cleanup for descendant processes. Record runner failures separately from task failures. The Python subprocess documentation explains process timeouts; those mechanics alone do not supply that execution boundary.
Then add task-specific negative controls. For a GPU kernel, use a candidate that returns correct-looking output for one shape but fails another. For a file-producing agent, test a missing artifact and a stale artifact from the previous run. Rerun the controls whenever the task contract or grader changes. A passing smoke test is a prerequisite for trusting a score, not proof that every way of gaming the evaluator has been eliminated.
A result record you can audit later
Keep the parent and candidate code IDs, exact model identity, evaluator version, input manifest, random seed, and environment with every run. Store the metric and raw evidence separately so a summary can be regenerated.
MLflow is one example of tooling for recording parameters, metrics, and artifacts. The record matters more than the tool choice: a simple manifest with durable files is better than a polished dashboard whose underlying run cannot be reconstructed.
{
"parent_commit": "<baseline>",
"candidate_commit": "<candidate>",
"evaluator_commit": "<read-only grader>",
"task_manifest_sha256": "<digest>",
"model_id": "<exact model or checkpoint>",
"seed": 42,
"budget": {"wall_seconds": 600, "max_tool_calls": 30},
"status": "pending",
"metrics": null,
"evidence_paths": []
}Promote a version, not a persuasive story
An example promotion rule is: all hard correctness checks pass, no required task family regresses beyond a predeclared tolerance, and the candidate improves the chosen quality-cost objective. The exact thresholds depend on the task. Write them down before seeing the winning candidate.
For a recursive claim, add an ablation: run the next improvement round with and without the changed component under the same budget. That asks whether the component helped further improvement, rather than merely improving one task.
If the result is inconclusive, keep it as a candidate. An archive can preserve useful alternatives without silently replacing the active agent. Publish failures and scope limits along with wins; they tell another researcher where the method is worth trying.
References
Turn the result into a repeatable experiment
Keep evaluator code, input manifests, and output evidence in the same research workspace. Start with the experiment records described in Meshia's docs.
Read about experiment records →