Speeding up tests while simulating or yielding

[Xrayez]

Version: 6.7.0+

Ability to simulate methods and yield for events during tests are quite handy. This of course leads to increased time for all tests to pass.

Recently I've figured a nice engine feature that allows to control how fast iterations are run from command line, see godotengine/godot#7269.

I've experimented with these settings: godot --fixed-fps 11 --disable-render-loop -d -s addons/gut/gut_cmdln.gd

Need to override this in addons/gut/gut_cmdln.gd if running with --disable-render-loop and to manually control drawing to my understanding (not required if the option is not present):

func _idle(delta):
	iteration(delta)
	VisualServer.force_draw()

The number for --fixed-fps is a bit weird, 11 is the minimum number that allowed me to pass all tests that actually depend on drawing (10/60 = 0.1666), using yield(get_tree, 'idle_frame'), should probably use yield(VisualServer, 'frame_post_draw').

This allowed me to speed up my tests by at least ~500% with reproducible results.

So I guess I'm asking whether anything could be added to GUT, or perhaps adding a wiki section would be enough. This certainly needs more thought.

[bitwes]

This is great. I've wondered if there wasn't something that could be done along these lines. I will add something to the wiki and to the command line help. I can't wait to try this out.

[bitwes]

Can you elaborate on "using yield(get_tree, 'idle_frame'), should probably use yield(VisualServer, 'frame_post_draw')"? I don't know anything about those signals or how you are using them. What cool magic are you pulling off with those?

[Xrayez]

yield(get_tree(), 'idle_frame') basically pauses method execution until next frame (iteration) is finished, which forces the rendering system to draw a frame for me. I kinda have to use this in tests because the basic painting module that I've written is dependent on some frames to be drawn before I can even test and assert anything.

Here some use case to make it more clear:

	# BRUSH PAINT =================================================
	paint.used_tool = Paint.Tool.BRUSH
	paint.brush.size = 64

	var brush_pos = paint.brush.global_position
	assert_eq(brush_pos, mouse_pos)

	paint.paint(true)
	yield(yield_to(get_tree(), 'idle_frame', 1.0), YIELD)
	paint.paint(false) # commit

	var image
	image = yield(paint, "rendered")
	assert_not_null(image)

	image.lock()
	var c = image.get_pixelv(mouse_pos)
	image.unlock()

	assert_ne(c, Color(0,0,0,0), 'Brush shoud paint')

I've discovered yield(VisualServer, 'frame_post_draw') as well which stops execution until a frame is drawn directly, which I think doesn't necessarily involve the whole game iteration to pass (for instance, when VisualServer.force_draw() is called) and this should probably make it more deterministic in terms of testing.

I think rendering is capped to 60FPS so this limits how fast these kind of tests could be run with --fixed-fps. Perhaps there's a way to increase rendering FPS to 120? Not sure. I'm considering separating those tests, those that depend on frames to be drawn and those that do not, so I could probably max out testing performance on the latter!

[bitwes]

Just wanted to get this down in the thread before I took off for the day. I still need to read the last post better to understand it. We have the start of a good wiki page here.

The fixed fps didn't help my game's tests much and caused some crashes. I did find the option --time-scale which makes everything run faster and has made a decent impact on execution time. Most of the yielding I do is for a X seconds so this helps a lot but does cause some tests to fail. I've tested values in my game up to 8 (1 is default). 8 caused a lot of failures (at least 10), but 3 only caused 1 test to fail of 1510. The test yields for some seconds so I can probably work around. Especially since it cut the time from 312s to 144s.

[Xrayez]

I tested --time-scale option and got similar results, the value of 3 allowed me to pass all tests, values higher than that make some tests fail (those that yield for >= 1 seconds). With --fixed-fps 11 yielding for timers works alright for me, yet still fails on drawing tests. For that matter, --time-scale option might not be as deterministic as it could be (for instance, see godotengine/godot#24334).

Note that for --fixed-fps to speed up execution, the value has to be lower than 60 to speed it up (so the value is actually inversed). Setting --fixed-fps to 0 should allegedly make it run without restriction (as fast as CPU allows), but seems like iterations themselves need to be controlled from script, but don't know how exactly yet.

To recap, some things like timers and rendering can limit the execution speed in order for the tests to have reproducible results. That's why --disable-render-loop is present and should be used in some cases, but most likely doesn't fit for unit testing.

[lawnjelly]

Just let me chip in here but saying that --fixed-fps is confusingly named imo. It is setting the variable fixed_fps in main.cpp, which is used to drive the physics tick rate. A better name would be --fixed-physics-delta, but even that isn't perfect.

I guess the original thinking behind the name is that it is giving the physics the impression that we are running at a fixed frame rate, but it obviously leads to confusion. There is a separate setting in Project->Debug->Settings->ForceFPS for actually setting a fixed frame rate I believe(!). :smile:

It looks like fixed_fps is actually only used in main_timer_sync.cpp:

	if (fixed_fps != -1)
		p_idle_step = 1.0 / fixed_fps;

Where the idle step is the time delta used to drive the physics ticks.

Example: if your physics tick rate is 60 ticks per second, and fixed_fps is 10, p_idle_step will be 0.1 seconds, and 6 physics ticks will be run on each rendered frame! I think, just from the code, that is untested.

[Xrayez]

@lawnjelly yeah I thought the same! You might as well report this to Godot repository too and I'd confirm that. Not too critical though but the name is indeed misleading.

[Xrayez]

I think this is documented somewhere already at this point, closing, not critical.