GdUnit failures: assertions vs cases

A test summary that says "2,552 failures" tells you nothing you can act on, unless you happen to know that the suite it came from had 3,502 test cases. It reads like the project is catastrophically broken. It is not. It is a number that was never counting what you assumed it was counting.

This is what happened in our Grid Placement test tree, and the fix is small enough to apply to any Godot project with a GdUnit4 suite.

What the number actually was

The full-tree run printed this:

total: 8 shards, 3502 cases, 0 errors, 2552 failures, 0 orphans, 0 aborted test bodies

2,552 failures out of 3,502 cases. You cannot fail 2,552 of 3,502 tests and still have a test suite that anyone runs. So either the tree was destroyed or the number meant something else. It meant something else.

The 8 shards are our setup, not the lesson: Grid Placement's tree runs 3,502 cases over a complicated system, so the runner fans out for speed. Everything below reproduces on a single shard — and on any stock GdUnit4 install.

GdUnit4's failures field counts failed assertions, not test cases. The console reporter sums every failure report collected during the run:

# godot/addons/gdUnit4/src/core/execution/GdUnitTestReportCollector.gd
static func count_failures(reports_: Array[GdUnitReport]) -> int:
	return reports_.filter(__filter_is_failure).size()

A test case is not the unit being counted. An assertion is. And a test case can contain a lot more than one of them.

Where the inflation came from

One suite accounted for most of it. The mode re-entry stress test drives 385 mode transitions, asserting on the current mode after each one:

# godot/test/grid_placement/e2e/mode_reentry_stress_e2e_test.gd
assert_int(transitions).is_equal(385)

A single failing case in that suite produces one failure report per transition it got through. That one shard reported:

Overall Summary: 347 test cases | 0 errors | 2458 failures

347 cases, 2,458 failures. That shard's real number was 11 failing cases.

This is not a GdUnit bug. A counter that reports assertion failures is doing what it says. It is a reporting bug — the runner was labelling the field "failures" and reading it as cases, and so did everyone who ran it, because the label is the only thing the reader has.

The fix: count distinct failing cases, print both

Two changes, both in the tree runner.

One, match the failing cases out of the log and de-duplicate them by suite and name. GdUnit reprints a failed case once per retry, so it goes in a set:

# scripts/testing/run_test_tree.py
FAILED_CASE_RE = re.compile(
    r"(res://\S+\.gd)\s*>\s*(.+?)\s*FAILED", re.MULTILINE
)

distinct_failing = 0
for result in results:
    distinct_failing += len(set(FAILED_CASE_RE.findall(result.get("plain", ""))))

Two, print both numbers, and let the label carry the distinction:

total: 8 shards, 3502 cases, 0 errors, 41 failing cases
      (2552 failed assertions), 0 orphans, 0 aborted test bodies

Same line, two figures, and now the one that answers "how much is red" is first. The per-shard reason got the same treatment, so it stopped calling assertions "cases" too.

The real figure across the tree is 41 failing cases out of 3,502 — about 1.2%. The other number was never wrong, just answering a different question.

Nothing about pass or fail changed. stats["failures"] still gates each shard, and the release gate still scans for the literal string it scans for. This is only a change to what gets printed.

Why the wrong number is worse than no number

A summary showing 2,552 failures does not get read. Nobody scrolls to the bottom of a wall of red to find the one new failure at the top of it. Within a few runs, the output becomes a ritual you perform rather than a report you read, and the moment something genuinely breaks, you are reading a wall of red looking for a difference that is not visible at that scale.

If your summary can report 2,552 failures and you cannot answer "is this worse than yesterday?", the summary is not carrying information. It is carrying volume, and volume is the one thing a reader learns to ignore.

The 41-case number is actionable on its own: it is a count you can watch move, put in a ratchet, and compare against a baseline recorded in a file.

The other half: known-failure lists rot without a resolution field

Fixing the count told us how much was red. It did not tell us whether the red was accounted for. So the next step was writing down the 41, and that is where the second trap is.

We keep a known_failures.txt listing every red suite on main, with a checker that fails if a listed suite has started passing (a stale entry is a false claim that a live regression is already known) and if an entry links no issue.

That is not enough. The first version of the file required only that each entry link an issue, and at the time it was written every one of the 13 entries was satisfied by the same umbrella issue: 41 failing cases across 13 red suites, one tracker. Then that issue closed as completed. The tracking was done; the failures were not fixed.

The finding was right and the closure was still wrong. One ticket cannot be the resolution claim for 41 separate failures — as soon as it closes, every one of those entries silently becomes "known and unowned". The release gate correctly reported every entry as a known failure whose tracking issue was closed. The check was working as designed. The data had rotted underneath it.

The fix is a field per entry that has to be checkable, not an intention. This is what the file looked like at the September 29 snapshot (b39c087), with 9 failing cases across 4 suites:

# Format: `<res://suite path> #<issue> resolve:<fix|investigate> #<n>` per line.
res://test/demos/integration/3d/placement_world_cross_mode_authority_e2e_test.gd #1429 resolve:investigate #1429
res://test/demos/integration/3d/demo_3d_drag_invalid_suppression_test.gd #1443 resolve:investigate #1443
res://test/grid_placement/e2e/mode_reentry_stress_e2e_test.gd #1444 resolve:investigate #1444
res://test/grid_placement/e2e/smooth_placement_e2e_test.gd #1445 resolve:investigate #1445

Note the count moved from 41 to 9 as the work landed. That is the point: the list is a ratchet, and a number you can watch fall is worth having.

Two values, and both have to mean something:

• resolve:fix #<pr> — an open PR that will make this pass. The checker reads the PR state. Once it merges, the entry is stale and rule 2 removes it.
• resolve:investigate #<issue> — the cause is genuinely not known yet. This is a promise to look, not permission to leave it red indefinitely.

Either way the entry names its own exit. An umbrella issue can still be the place the work is discussed; it just cannot be the thing that keeps the entry satisfied.

Takeaways

• Label the unit. "2,552 failures" next to "3,502 cases" is a reporting bug, not a testing bug. Count the thing your reader is asking about.
• Print both numbers when both are true. A failed-assertion count is real information. It is just not the number that answers "how much is red".
• A known-failure list needs a per-entry resolution claim, and that claim has to be machine-checkable. "Is the tracking issue open?" is satisfiable by one ticket forever, which is how a red tree ended up owned by nobody.
• Keep the baseline in a file the gate reads, not in a constant in a script. It has to be re-measurable and it has to fail closed when missing.

None of this needs Grid Placement. GdUnit4 is not the only runner that reports assertions where a reader expects cases; check the one you use by running it over a suite with a known, small number of failures and seeing whether the headline number matches.

Common mistakes

• Trusting the headline number's label. Read the field the runner counts, not the name the runner gives it.
• Counting failure lines instead of failing cases. A retried failure is reprinted. De-duplicate by suite and case name.
• Breaking the gate while fixing the report. Changing what is printed is safe; changing what gates the shard is a different change with a different review.
• Treating one issue as the resolution for many failures. Give each entry its own exit.