Find blank frames in Godot test captures

If your pipeline records evidence about a visual result, the evidence needs to be able to say that the result was not there. Otherwise you have built a very reliable machine for photographing nothing.

This applies to any automated screenshot or video-proof setup — Godot, Unity, Playwright, Cypress, whatever. The specific tooling here is ours, but the pattern is general and the failure it prevents is quiet.

The failure: a resized, unredrawn backbuffer

The frame you capture is not necessarily the frame you meant to capture. In Godot, resizing a window or viewport updates the backbuffer before the scene redraws into it. Grab the texture at that moment and you get a correctly-sized image that is a single flat colour.

The nasty part is that nothing about that image is wrong. It has the right dimensions. save_png returns OK. The file exists, the byte count is reasonable, and it hashes perfectly reproducibly — run it tomorrow and you get the identical SHA-256. Every downstream check that asks "did the capture succeed?" says yes.

It is a perfectly reproducible hash of nothing.

The check

Refuse to save a frame in which nothing was drawn. Sample a coarse grid of pixels and compare each against the top-left; if every sample matches, nothing rendered:

static func is_rendered(p_image: Image) -> bool:
	var first: Color = p_image.get_pixel(0, 0)
	var step: int = maxi(1, int(p_image.get_height() / 32))
	for y: int in range(0, p_image.get_height(), step):
		for x: int in range(0, p_image.get_width(), step):
			if not p_image.get_pixel(x, y).is_equal_approx(first):
				return true
	return false

A full pixel sweep is unnecessary and slow; the question is only "is there any variation at all", and a coarse sample grid answers it. Then the save path refuses rather than writing:

static func save(p_image: Image, p_path: String, p_label: String = "") -> bool:
	if p_image == null:
		return false
	if not is_rendered(p_image):
		push_error("FrameSave: %s is a single flat colour (%s) at %dx%d — nothing rendered" % [
			p_label if p_label != "" else p_path,
			p_image.get_pixel(0, 0).to_html(false),
			p_image.get_width(),
			p_image.get_height(),
		])
		return false
	var err: Error = p_image.save_png(p_path)
	if err != OK:
		return false
	print("FRAME_SAVED %s sha256=%s" % [p_path, FileAccess.get_sha256(p_path)])
	return true

The caller treats a false as a hard failure and exits non-zero. That is the whole mechanism: a capture that proves nothing does not get recorded as proving something. (Quote abbreviated: the shipped version also enforces an expected-size contract on the image.)

Route every producer through one path

The check only helps if it cannot be bypassed, which means the save itself has to live in one place. We have a shared FrameSave helper with exactly one job — grab, refuse if flat, write — plus a Proof3DDriver base carrying the same check for the 3D proof flow.

Measured on our smoke tree at origin/main as of 2026-09-30, using the rule that a driver counts as an evidence producer only if a code line persists a PNG (not a doc comment mentioning one):

Measure	Before	Now
Smoke drivers (non-test .gd)	52	52
Evidence producers (persist a PNG)	35	35
Reach a flat-frame refusal	11	28
Hand-roll the inline framebuffer grab	11	1

Coverage went from 11 of 35 to 28 of 35, and the drivers still hand-rolling their own grab went from 11 down to 1. Those counts move as the tree does, so read them as of the date above rather than as standing facts.

Two limits worth knowing about

This is the part most write-ups skip, and it is the part that matters if you are deciding how much to trust a green check.

A guard only sees what it scans

We have a CI script that enforces the contract — every driver that writes a PNG goes through a base. It passes in advisory mode and it passes under --strict.

It does not see one of our evidence paths. Five demo drivers route through a shared proof harness that has its own capture_named(), which grabs the framebuffer and calls save_png itself, with no flat-frame check anywhere in the file. That is 31 PNGs recorded as evidence through exactly the failure mode this pattern exists to prevent:

Driver	PNGs saved through the harness
demo_3d_structure_proof	13
demo_3d_top_kit_proof	6
demo_smooth_3d_capture	5
demo_3d_fence_proof	4
demo_smooth_3d_socket_capture	3

The guard could not see it because its scan root was godot/test/smoke/, and the harness lived in godot/test/support/. At the September 30 snapshot the check passed --strict while 31 PNGs went through unguarded. That hole is closed on main now: the harness routes through FrameSave.grab and FrameSave.save, so a flat frame is refused the same way everywhere else. The lesson survives the fix, and it is worth naming rather than rounding off: a guard's coverage is bounded by its scan root, and a green guard tells you nothing about anything outside that root.

The lesson is not "our guard was badly written." It is that a guard's coverage is bounded by its scan root, and a green guard tells you nothing about anything outside that root. When a check reports success, measure what it did not look at. The number of producers it can see is the denominator that matters, and it is rarely the number you assume.

An exemption justified by a false reason is worse than no exemption

The same guard exempts one driver from the contract with the comment "writes a raw .rgb stream, not a PNG frame". Measured, that file contains 2 save_png calls, 13 .png filenames, and 2 hand-rolled framebuffer grabs, with zero flat-frame refusals. It is the one driver the guard formally excuses, and the stated reason is why nobody looked. Treat that as a floor rather than the full count: the guard also allows any file whose code merely names a base class, so files that use a base only for its unrelated input helpers slip through the same way. Naming a base is not the same as routing pixels through it.

An exemption is a claim about the code. When it is wrong, it does not merely fail to help — it actively suppresses the signal, because a reviewer reading the exemption takes the claim at face value and moves on. If your guard has exemptions, verify the reason against the file, the way you would verify any other assertion.

Takeaways

• Evidence about a visual result must be able to describe its absence. A hash of a blank frame is a perfectly reproducible hash of nothing.
• Refuse before writing, and make the caller fail. A false return that the driver treats as a hard error is the mechanism; a warning is not.
• Put the save in one shared place so the check cannot be skipped by accident, then measure how many producers actually route through it.
• Audit the guard's scan root, not just its verdict. Our own check passes --strict while missing 31 PNGs outside the directory it scans. Green means "nothing wrong inside the boundary," not "nothing wrong."
• An exemption is a claim. Verify it against the file, because a false reason reads as an answer and nobody looks twice.

None of this is Godot-specific. Any pipeline that captures a rendered frame can be handed an empty backbuffer, and the hash will be stable either way.

Common mistakes

• Checking the exit code instead of the contents. save_png returning OK says nothing about whether anything was drawn.
• Hashing the artefact as proof. A hash proves reproducibility. A blank frame is reproducible.
• Sampling one pixel. Sample a grid; the first pixel matching itself proves nothing.
• Writing the check once and leaving a second path open. Two save paths is one covered path and one unknown.
• Treating a passing guard as coverage. Count what it did not scan.