easybuild-framework

easybuild-framework copied to clipboard

Replaces https://github.com/easybuilders/easybuild-framework/pull/4052

After discussion with @boegel we've decided that the the approach in #4052 is not maintainable. By pushing the loop over the entries in an EasyStack file to main.py, and by basically re-executing part of main.py for each item in the EasyStack file, we can properly reinitialize the configuration options for each item (if there are EasyConfig-specific options defined). This has the advantage that we can reuse the existing machinery of things like set_up_configuration.

It also means that EasyBuild builds a dependency tree per item in the EasyStack file. On the one hand, that is desirable since two items might have different --from-pr arguments, and thus the same dependency A.eb might in theory resolve to two different PRs depending on the options passed for that item in the EasyStack file. The downside is that since we don't have a single dependency graph, we cannot e.g. submit a single set of jobs with --job: we'll need to make sure EasyBuild becomes aware of the order in the EasyStack file, and make sure that jobs for later items wait for all jobs of earlier items. That's something we'll implement later though...

Note that this is really a work in progress, a lot more functions need to be put in place before this properly works...

TODO for tests (I'll try to look into that):

• add test for set_up_configuration (which also covers reconfigure)
• add test for new dict_to_argslist function
• add end-to-end test with easystack file that sets options for some easyconfigs + verify result (using toy installs)

TODO for tests (I'll try to look into that):

• add test for set_up_configuration (which also covers reconfigure)
• add test for new dict_to_argslist function
• add end-to-end test with easystack file that sets options for some easyconfigs + verify result (using toy installs)

We should probably also have an end-to-end test that tests the behaviour of options being passed on both command line and in EasyStack file. E.g. pass a --hidden and a --disable-hidden on the command line and in the EasyStack file respectively. The resulting module should not be hidden, as the intended behaviour here is that options from the EasyStack file 'win'.

Edit: done in https://github.com/easybuilders/easybuild-framework/pull/4057/commits/725615640c137fa9b4f04fe27f0457e70f091b7e

Todo, update documentation: https://github.com/easybuilders/easybuild/blob/develop/docs/Easystack-files.rst After that is done, we can merge this PR

edit (by @boegel): see https://github.com/easybuilders/easybuild/pull/832

Merging #3749 caused a merge conflict here, which I've fixed just now.

I'll propose some tweaks to the docs update in https://github.com/easybuilders/easybuild/pull/832, so we can get this merged...