We added a gate to catch tests that never run

Your pipeline is green. Did every test actually run? In October 2026, a DynoInvoice merge request passed two green pipelines even though 16 of its 17 negative tests had been skipped.

The fix took more than changing the tests. We added a gate to compare the test cases reported by the runner with the test methods in the source, so the same kind of miss is harder to overlook.

Sixteen tests, one reported case per class

The merge request was for a database library in DynoInvoice, the e-invoicing system agents are building with Keel. Its task called for negative tests covering stale versions, concurrent updates, another tenant’s rows, missing transactions, and row-level security on and off. The coding agent wrote 17 tests in a shared base class.

The tests were written in Kotlin like this:

@Test
fun `rejects a stale version`() = withService {
    assertThrows<StaleVersionException> { store.update(staleRow) }
}

withService returns the value of its lambda. Here, that value is the exception returned by assertThrows, so the test method returns a value instead of Kotlin’s Unit.

JUnit Jupiter only runs test methods that return void (in Kotlin, Unit). It skips the others without failing the build. The pipeline was green, listed six test classes, and each class had run one case.

The fix was to use a block body, so the method returns Unit:

@Test
fun `rejects a stale version`() {
    withService {
        assertThrows<StaleVersionException> { store.update(staleRow) }
    }
}

The first review missed it

The independent reviewer found a real gap: a missing batch-audit test. But the review did not catch that most tests were not running. Nor did a check that required each changed test class to have at least one passing case. Each class had exactly one.

A separate agent noticed later while working on something else. It opened the test report, saw one case per class, and compared that with the 17 @Test methods in the shared base class. Re-running one class reported two tests.

An independent agent caught it before merge. The review round and the existing check both missed it.

Make the count part of the gate

Fixing the 16 methods addressed this merge request. It didn’t address how the same mistake could slip by again. So we made two changes:

• Every @Test method must return Unit.
• A new gate compares the number of test cases in each class’s report with the number of @Test lines in its file.

The gate has tests of its own: an expression-bodied test must make the count come up one short, and a missing class must count as zero. After the fix, every case in the six classes ran, with no failures or skips. The change merged on 18 October 2026, in commit daaaaabd in the public repository.

Checks worth borrowing

Read the case count, not just the colour. Green means the checks that ran passed. It doesn’t tell you whether all the expected cases ran.

Compare the report with the source. For each class, check that the reported case count matches the number of test methods. Adapt the comparison for inherited and parameterised tests, then deliberately add a skipped test and confirm the gate fails.

Prove negative tests can fail. Break the behaviour a test is supposed to protect and check that it goes red. Otherwise, another layer may be rejecting the input and making the test look effective when it isn’t.

Here is a minimal starting point for Kotlin tests with Gradle’s JUnit XML reports:

# fail if any test class ran fewer cases than it declares
for src in $(grep -rl --include='*.kt' '@Test' src/test); do
  cls=$(basename "$src" .kt)
  declared=$(grep -cE '@Test([^A-Za-z]|$)' "$src") # not @TestInstance
  ran=$(grep -ho "testsuite name=\"[^\"]*\.$cls\" tests=\"[0-9]*\"" build/test-results/test/*.xml \
    | grep -o 'tests="[0-9]*"' | grep -o '[0-9]*' | head -1)
  [ "${ran:-0}" -ge "$declared" ] || {
    echo "$cls: declared $declared, ran ${ran:-0}"
    exit 1
  }
done

What Keel did

Keel’s records tie a task’s tests to its acceptance criteria. The coding agent can’t change those criteria, and the server marks a task done from the merged change and its CI verdict. On DynoInvoice, our rules forbid weakening a check to get green, and an independent reviewer agent checks every merge request, including test changes. Here, that review missed the skipped tests. After the miss surfaced, the team added a case-count gate to the build.

We’re opening Keel to design partners building with coding agents. Design partners get Keel free during the beta, with hands-on setup and onboarding. Apply at dynokore.ai.

Written by the DynoKore team with AI assistance. First published on dynokore.com.