# Mutation testing for AI-generated code

> Why coverage cannot check tests that an agent wrote, how surviving mutants can, and the rules Reapr's mutation-audit agent skill follows.

Source: https://reapr.dev/guides/mutation-testing-ai-generated-code

Teams now ship a lot of code that an agent wrote, often with tests the same agent wrote. The tests pass and
coverage is high. Neither fact says the tests would catch a bug.

## Coverage measures execution, not verification

Line coverage records which lines ran while the tests ran. It cannot tell a test that checks the result from
one that only calls the function:

```ts
it("computes the shipping fee", () => {
  expect(shippingFee({ total: 120 })).toBeDefined();
});
```

That test covers every line of `shippingFee`. Change `total > 100` to `total >= 100`, or return `0` instead of
the fee, and it still passes.

## What a mutation score measures

A mutation engine such as StrykerJS changes the code on purpose, one small change at a time: `>` becomes `>=`,
a condition becomes `true`, a return value changes. Each change is a mutant. If a test fails, the mutant is
killed. If every test still passes, the mutant survived, and you have a concrete change in behavior that no
test catches.

The score is detected / (detected + undetected): killed and timed-out mutants over those plus the survived and
uncovered ones. It is the share of the changes your tests would notice.

## Why this fits agents

A surviving mutant is a precise, checkable task: this line, this change, no test failed. Reapr prints each one
in its JSON output:

```json
{ "file": "src/a.ts", "line": 10, "column": 3, "mutator": "ConditionalExpression",
  "replacement": "false", "status": "Survived" }
```

An agent can write a test for that exact behavior and prove it worked: run Reapr again and check that the
mutant is now killed. The evidence comes from the engine, not from the agent's own judgment.

## The rules that keep it honest

Every shortcut that raises the score can make the tests worse. Reapr's
[mutation-audit agent skill](https://github.com/reapr-corp/reapr/blob/main/plugins/reapr/skills/mutation-audit/SKILL.md)
writes these down as hard rules:

- Never change production code to kill a mutant. If a test shows the code is wrong, report it.
- Never edit engine config, thresholds or Reapr flags to change the result, and never create an engine config.
- Never delete or weaken tests. Change an existing test only with the human's OK.
- One behavior per test, in the repo's conventions, inside the diff scope.
- Assert intended behavior, not current behavior. When the code disagrees with the docs, types, names or
  callers, ask the human instead of writing a test that pins a possible bug.
- A test is done only when a later Reapr run shows that it kills a mutant. Equivalent mutants, which no test
  can kill, are listed for the human instead of getting a forced test.

The skill also lists tests that cover the diff but kill no mutant, and then checks the diff for edge cases that
mutation operators do not produce: boundaries, empty input, error paths, time zones and numeric precision.

## What the skill's evals showed

The skill was tested on a throwaway TypeScript + Vitest + StrykerJS repo, with and without the skill
([pull request #14](https://github.com/reapr-corp/reapr/pull/14)). On a weak-boundary fixture, the run with the
skill took the score from 61.90% to 90.48% and passed 7 of 7 checks. The run without it reached a higher score
but rewrote an existing test without asking. With no Stryker config in the repo, the skill stopped and pointed
to `npx stryker init`. Without the skill, the agent wrote its own config and then changed tests.

## Try it

The skill has its own page with the install commands and results: [reapr.dev/skill](/skill).

```sh
curl -LsSf https://github.com/reapr-corp/reapr/releases/latest/download/rpr-installer.sh | sh
rpr run --diff origin/main --json
```

In Claude Code, install the skill with `/plugin marketplace add reapr-corp/reapr` and
`/plugin install reapr@reapr`, then ask it to check the tests on your branch. For other agents, point them at
[AGENTS.md](https://github.com/reapr-corp/reapr/blob/main/plugins/reapr/AGENTS.md).

## Frequently asked questions

### Is code coverage enough to trust AI-written tests?

No. Coverage records which lines ran during the tests. A test that calls a function and asserts nothing useful still covers every line it runs. A mutation score checks whether a test fails when the code changes, which is what a test is for.

### How can an agent use mutation testing?

Run rpr run --diff origin/main --json. Each surviving mutant has a file, line, column, mutator and replacement. The agent writes one focused test per survivor, runs Reapr again, and keeps the test only if the mutant is now killed.

### Should an agent change code to kill a surviving mutant?

No. A surviving mutant is evidence about the tests. If a new test shows the code is wrong, that is a bug for a human to decide on. Reapr's mutation-audit skill never changes production code, engine config or thresholds.