MUSE_OS icon indicating copy to clipboard operation
MUSE_OS copied to clipboard

Add script for regenerating data in `tests/example_outputs`

Open alexdewar opened this issue 1 year ago • 3 comments

This is partly a small QoL improvement, but having a way of regenerating this data programatically is also a prerequisite for making it possible to run regression tests without having to store the results in the git repo (see #351).

Are we happy that this makes sense regardless of what solution we come up with for #351? @dalonsoa @tsmbland

alexdewar avatar Jun 21 '24 11:06 alexdewar

I'm not sure, to be honest. The whole point of regression tests is that new data created with the latest version is the same than the one created with a previous one, and if it changes, it is for a good, understood reason. If we are creating data programmatically, it does not make sense to use that data in regression tests, and it will be using the latest (potentially wrong) version of the code and not the older (in principle correct) version.

Am I missing anything?

dalonsoa avatar Jun 24 '24 12:06 dalonsoa

I think the point is that you'd only run this if you've intentionally made a change to the model that you know will change the results. Just saves you from having to run all of the example models individually to update the results files.

Or, it could be part of a more complex CI workflow like the one that Alex suggested in #351, but in this case it will only be run with old versions of the code which are known to be correct

tsmbland avatar Jun 24 '24 12:06 tsmbland

Ah, ok. In the first use case it makes sense to have an automated script, yes. I still haven't made my mind on the best approach for #351 , to be honest.

dalonsoa avatar Jun 24 '24 12:06 dalonsoa