Gut
Gut copied to clipboard
bitwes
Run tests in the same order everytime
The order of the following can vary from run to run (due to Godot):
- Files in a directory
- Inner classes in a file
- method names in a class/file
This means that the tests are executed in a different order just about every run. This can make comparing output from one run to the next difficult.
It would be best if the test were run in this order:
- Alphabetical by file name
- line number for Inner classes
- line number for methods
If this isn't feasible then alphabetical for file, inner class, and method should be possible.
If you want to ensure that you aren't building in some test order dependencies then the user should be able to specify a seed to randomly order the files/inner classes/methods.
If you want to ensure that you aren't building in some test order dependencies then the user should be able to specify a seed to randomly order the files/inner classes/methods.
I'd also like an option to not specify a seed so the RNG orders tests differently every time i run them.
The next release of GUT has an xml export and some hidden away json export logic. This should make comparing results easier. I'm leaving this open because I still want it but I'm taking it out of the 7.2 milestone.
Well, looks like I missed this 7.2 milestone train by a few hours. Anyways, I figured I would submit a pull request to this issue that does ordered execution, based upon how they are declared in the script file itself.
I added a new function, and a Class to utils.gd, that allows the script methods to be parsed, and found in the source code for the script, and define the order of return based upon where they have been declared in the script. This will ensure that scripts will always execute in the same order, based upon how they were declared in the test script file itself. The function does not recurse into base scripts, as that adds another level of complexity to it, but will work with the target script file in question.
Also, with the new parsed methods function, no longer need to test for the test_prefix, as it's handled by the function, all that is left, is to create the Test class instance, with the data, and append it to the test_script.tests variable.
If you have any questions, let me know.
