Why Stryker's incremental score covers the whole package

Updated . Numbers from the Reapr bench report. Read this guide as Markdown.

You run Stryker on the lines you changed and it reports a score for hundreds of mutants you never touched. That is not a bug in your ranges. It is how incremental mode builds its report.

What happens

With incremental: true, Stryker keeps the results of earlier runs in an incremental file and reuses them. When you also pass --mutate with line ranges, it tests the mutants in those ranges and carries the earlier results for everything else into the same report. Stryker’s incremental docs say so directly: mutants not in scope are not removed from the full mutation report.

The score at the bottom of that report is then the score of the package, mixed from today’s run and older ones.

A measured example

In Reapr’s measurements on one package of a pnpm + Turborepo monorepo (~55k files incl. build caches, StrykerJS 9.6.1, Vitest 4), a plain npx stryker run --mutate <ranges> on the diff’s ranges reported 87.0% with 38 survivors, across the package. The diff itself had 3 mutants, all killed: 100%. Both numbers were correct; they answer different questions. Source: the README’s measurements and the bench report.

A reviewer who reads 87.0% thinks the change left 38 gaps. It left none.

How to get the score of the diff

Filter the JSON report to the mutants inside the changed ranges, then compute the score over them:

CompileError, RuntimeError, Ignored and Pending do not count. This is the mutation-testing-metrics definition Stryker uses, so the result compares one to one with Stryker’s own numbers.

“Inside the range” needs care. Stryker counts a mutant as in a range when it starts on or after the first column of the range’s first line and ends on or before its last line, comparing line and column together. Use the same rule, or your count will drift from Stryker’s at the edges.

How Reapr does it

rpr run --diff origin/main reads the diff, runs Stryker on those ranges with your config (incremental settings included, untouched), and filters the report with Stryker’s containment rule. Results that Stryker carried over from outside the diff are excluded, and a warning says how many:

warning: stryker report carried 12 results from outside the diff scope (incremental); excluded from the diff score

The score, the counts and the survivors in Reapr’s output are for the diff only. Stryker’s unfiltered report is still in the run record, .git/reapr/runs/<run_id>/mutation.json, if you want the package view too.

In the same earlier measurements, Vitest 4 with bail on counted some killed mutants as survived: the score was 87.8% with bail on and 98.0% with disableBail: true, on the same code. The effect depends on the setup: an independent check on another Vitest 4 repository saw no difference. It can happen, so check it on yours: run once with disableBail: true and compare the scores. Reapr warns when your config uses the Vitest runner with Vitest 4 or later and disableBail is not true. It only warns; whether to set disableBail: true is your call.

Questions about this guide

Why does Stryker report hundreds of mutants when I only mutated a few lines?
With incremental true, Stryker reuses results from its incremental file and carries results from outside the --mutate ranges into the report. Its docs say mutants not in scope are not removed from the full report. The score in the report is then for the package, not for your lines.
How do I get the mutation score of only the changed lines?
Filter the report to the mutants whose location falls inside the changed ranges and compute detected / (detected + undetected) over them. Reapr does this with rpr run --diff, using Stryker's own containment rule, and warns how many carried-over results it excluded.
Should I turn off incremental mode?
Not necessarily. Incremental mode is a cache and can save time on repeat runs. The problem is only the score in the report. Read the score of the diff from a tool that filters it, or filter the JSON report yourself.