easybuilders
Implementing hidden modules through module-hide/hide{}
Hi,
I started using easybuild more recently, and I noticed that the implementation for hidden modules through the easyconfig parameter hidden or through --hide-deps is done through the preceding . on the modulefile.
Has there been any consideration in using the more-powerful module-hide from EnvironmentModules or the equivament hide{} from Lmod?
I guess the implementation of our support for hiding modules pre-dates the support in the module tools?
I think it makes sense, and perhaps we can auto-opt-in to using (module-)hide if the modules tool being used is sufficient recent.
Another option could be adding a configuration option like --module-hiding-by which takes options like dot-prefix or native-support. That way people can opt-in to using the "modern" support in module tools, without us changing default behaviour of EasyBuild, since doing so without bumping major version may surprise people...
My two-cents for those who may read this issue: while support for hidden modules could indeed be improved in EasyBuild, I still don't believe this is the correct way to do this. A build-time decision about whether a module is visible or not means that a change of heart requires the software to be rebuilt (or at the very least --module-only --force...which we know can be brittle between EasyBuild versions due to changes in EasyBlocks or general behaviour).
I think this is better handled in a more dynamic way by the hook mechanisms of the module tools themselves. For Lmod see https://lmod.readthedocs.io/en/latest/170_hooks.html (a relevant specific example: https://github.com/TACC/Lmod/blob/main/contrib/more_hooks/SitePackage.lua#L177-L196), and for Environment Modules see https://modules.readthedocs.io/en/latest/module.html#hooks
I very much agree with @ocaisa on this. I probably would go even further by suggesting we remove all support for hiding modules in EasyBuild and push people to directly use the modules tools for this. The bonus here is that we remove the need to maintain and update the code for hiding modules and this simplifies our life when the modules tools update their mechanisms for doing this.
Couldn't the brittleness and the need to reinstall modules mentioned by @ocaisa be fixed by using the ModuleRC EasyBlock, i.e. just like we consider "default" a separate recipe, this kind of rules could also be a separate recipe.
Maybe a concrete case would help frame the discussion.
I was trying to install all the currently 6 supported foss toolchains (rationale: since GCC, FFTW, and some others take some time to compile, I'm hoping having these "pre-installed" would help me kickstart future installations that depend on these). That totals in 339 modules IIRC. They need to be there, but my users will not load M4, Autoconf, libxml, etc. I think having a "clean" module avail with just 6 modules for the toolchains instead of 339 modules flooding the terminal is very nice.
Now imagine next I want to make the Python modules available/visible. Actually, these are already installed by the previous process and hidden. My expectation on using eb to install them to the same prefix would be that nothing would be compiled, the module files would just be made visible. I know it's a high expectation, but what happened was that the hidden dot-prefix modules were completely ignored (the idempotency cares for the prefix) and all the Python modules were re-compiled and new visible module files were created for them (which were exactly the same as the dot-prefix ones). This is basically what @ocaisa 's refers to as an issue with the current hidden-deps method.
I don't know what's the correct way to address this. Maybe I'm misunderstanding the objective of easybuild, since I thought the "configuration" of the modulefile corresponding to the build was also part of the objective.
Maybe I'm wishing easybuild to do things it cannot do 😅
I agree with @ocaisa and @branfosj here: deciding which modules to hide should not be done at installation time, and hence not in EasyBuild.
It's better to have separation of concerns, and use the features supported by the modules tool to determine which modules are visible to end users, and which are not.
I wonder what @xdelaruelle, @rtmclay, @mrcawood think about this (as maintainers of different environment module tool implementations).
I have a strong preference for hiding modules with module-hide command rather prefixing module file name with a dot. If it is later decided to hide or not to hide anymore a module, using the modulerc command avoids a modulefile rename that leads to update every other modulefiles requiring or conflicting with it. Using dot prefix also breaks advanced version search capabilities (e.g., module load foo@3:)
Using the perspective of a sysadmin using EasyBuild to install my software stack, I would love the build tool to handle the modulefile hiding work. Having to handle it myself afterward is tedious, especially that many installation throws 100+ modulefiles with only one of these really useful for my end users.
As it was clearly expressed here, end users only want to see the leafs of our module trees. If dependencies are visible it makes their search to find what they could use on the machine harder.
I have worked on such feature for Spack few years ago: if hide_implicit is enabled, a .modulerc file is generated with module-hide command for each non-leaf generated modulefiles. Later on you can unmask some of the dependencies by asking install of them: nothing is reinstalled, but the .modulerc file is updated to remove the module-hide command for this module, as you declare it a leaf in your tree with this direct install order.
