KoBeWi
Add automated tests
This is a project that benefits a lot from reliability, so I think it will be good to add automated tests to better ensure that. There's some difficulties due to it modifying a Godot project and doing some unusual things, but I think it could be done well enough. And if that turns out to be a barrier, then partial test coverage would still be better than no test coverage.
I've been doing some manual testing, so I figure I should automate the testing where I can. But since I won't be maintaining them in the long run, I should ask you about it. Do you have any preferences for an automated testing setup? I think GUT and gdUnit4 would both be adequate.
My idea of a good and easily maintainable automated integration test suite is to have just a few types of tests that share the same structure, and then to cover the most important functionality with these. Then don't worry about testing everything.
After a little look at the code, the types of tests I'm thinking of focusing on are these:
- Set up global Data and then run the Execution scene. Instead of actually changing the project folder or doing other things, add to a human-readable String array of actions (such as "move file1 from root to subfolder") that change the project folder. Then this can be checked against an expected order of actions.
- Set up a routine for execution in the Main scene and then check that the resulting global Data when about to enter the Execution scene is correct.
I'm not really opposed to having tests, but I never did them outside the engine itself and I'm not familiar with GDScript testing solutions. Also I don't know if I'll be able to maintain them. My only preference would be to make the tests least invasive as possible, perfectly limited to a single top-level directory.
Though I think their usefulness would be mostly limited to tasks. Not sure if you can test GUI elements (e.g. to catch bugs with managing routines).
Instead of actually changing the project folder or doing other things, add to a human-readable String array of actions (such as "move file1 from root to subfolder") that change the project folder.
Wouldn't that be unreliable in itself? If you mock the behavior, you risk that the tests will only ensure that mocked actions are correct. (or maybe I misunderstood something)
I made an example test which I'll pull request together with this post. Let me know what you think.
Wouldn't that be unreliable in itself? If you mock the behavior, you risk that the tests will only ensure that mocked actions are correct.
That's correct. The idea was to separate the logic from API calls so that the only unreliable part would be assumptions guaranteed by the API, but looking at the code more closely this would seem to require a complete rewrite. So I can't recommend it anymore, but at least in theory it should be a fine way to test a project like this. Or maybe it was just a bad idea.
Not sure if you can test GUI elements (e.g. to catch bugs with managing routines).
Testing the GUI can be done. One way is to find the buttons and call something like simulate_mouse_press_at(button.get_global_rect().get_center()). Another way is to save a screenshot and put Marker2D nodes over where the buttons are, then simulate presses on them. Both of these come with some difficulties and maintainance issues though.
I think this can be closed now? Or you could add a tracker for missing tests.
Sure, the basic point of this issue is done. I'm not sure where exactly to draw the line between tests worth making and not worth making so I won't make a tracker.