Using `dependencies` instead of `hiddendependencies` leads to endless loop

[Flamefire]

I had a module which depends on a hidden module. Like:

CUDAcompat-11.6-510.73.08.eb:

name = 'CUDAcompat'
version = '11.6'
nv_version = '510.73.08'
versionsuffix = "-" + nv_version
hidden = True
toolchain = SYSTEM

CUDAcompat-11.6.eb:

easyblock = 'ModuleRC'

name = 'CUDAcompat'
version = '11.6'

homepage = 'https://docs.nvidia.com/deploy/cuda-compatibility/index.html'
description = "description"

toolchain = SYSTEM

dependencies = [(name, version, "-510.73.08")]

moduleclass = 'system'

Now building the former with --robot results in an endless loop eventually exiting with

ERROR: Maximum loop cnt 10000 reached, so quitting (easyconfigs: [{'short_mod_name': 'CUDAcompat/11.6', 'full_mod_name': 'CUDAcompat/11.6', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2addbd0beb90>, 'dependencies': [{'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/EasyBuildDev/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6.eb', 'builddependencies': []}, {'short_mod_name': 'CUDAcompat/.11.6-510.73.08', 'full_mod_name': 'CUDAcompat/.11.6-510.73.08', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2addbd1556d0>, 'dependencies': [], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/EasyBuildDev/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb', 'builddependencies': []}], missing_easyconfigs: [])

In the log I see the following repeated many times: robot.py:433 INFO Robot: resolving dependency {'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False} with /i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb

Some more log:

== 2022-07-29 10:57:07,409 robot.py:404 DEBUG Looking for easyconfig for {'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}
== 2022-07-29 10:57:07,409 easyconfig.py:2156 DEBUG Obtained easyconfig path from cache for ('CUDAcompat', '11.6-510.73.08'): /i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb
== 2022-07-29 10:57:07,409 robot.py:433 INFO Robot: resolving dependency {'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False} with /i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb
== 2022-07-29 10:57:07,411 one.py:548 DEBUG retrieve_blocks_in_spec: derived easyconfig format version: 1.0
== 2022-07-29 10:57:07,411 easyconfig.py:2249 INFO Contents of /i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb verified against easyconfig filename, matches {'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}
== 2022-07-29 10:57:07,412 robot.py:445 DEBUG Added {'short_mod_name': 'CUDAcompat/.11.6-510.73.08', 'full_mod_name': 'CUDAcompat/.11.6-510.73.08', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec75673f50>, 'dependencies': [], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb', 'builddependencies': []} as dependency of {'short_mod_name': 'CUDAcompat/11.6', 'full_mod_name': 'CUDAcompat/11.6', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec754dded0>, 'dependencies': [{'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6.eb', 'builddependencies': []}
== 2022-07-29 10:57:07,412 robot.py:452 DEBUG Unprocessed dependencies: [{'short_mod_name': 'CUDAcompat/11.6', 'full_mod_name': 'CUDAcompat/11.6', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec754dded0>, 'dependencies': [{'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6.eb', 'builddependencies': []}, {'short_mod_name': 'CUDAcompat/.11.6-510.73.08', 'full_mod_name': 'CUDAcompat/.11.6-510.73.08', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec75673f50>, 'dependencies': [], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb', 'builddependencies': []}]
== 2022-07-29 10:57:07,412 tools.py:121 DEBUG Finding resolved modules for [{'short_mod_name': 'CUDAcompat/11.6', 'full_mod_name': 'CUDAcompat/11.6', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec754dded0>, 'dependencies': [{'external_module_metadata': {}, 'short_mod_name': 'CUDAcompat/11.6-510.73.08', 'toolchain': {'version': 'system', 'name': 'system'}, 'name': 'CUDAcompat', 'full_mod_name': 'CUDAcompat/11.6-510.73.08', 'system': True, 'build_only': False, 'versionsuffix': '-510.73.08', 'version': '11.6', 'toolchain_inherited': True, 'hidden': False, 'external_module': False}], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6.eb', 'builddependencies': []}, {'short_mod_name': 'CUDAcompat/.11.6-510.73.08', 'full_mod_name': 'CUDAcompat/.11.6-510.73.08', 'ec': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x2aec75673f50>, 'dependencies': [], 'hiddendependencies': [], 'hidden': False, 'spec': '/i/easybuild-easyconfigs/easybuild/easyconfigs/c/CUDAcompat/CUDAcompat-11.6-510.73.08.eb', 'builddependencies': []}]

So it seems to resolve the module with the EasyConfig but ignoring the hidden property but when then matching the spec it uses the hidden property so it searches for the same file again and again. At least there should be an error. Better would be to just ignore the hidden property for the resolver matching

[ocaisa]

I think this is a side effect of the MNS used not requiring parsing of the easyconfig to resolve dependencies (hierarchical MNS does require parsing so probably would work in this case). The MNS assumes that it can construct the module name but it actually can't due to the hidden = True which it would only know about if it parsed the actual file.

Introducing parsing for the default MNS would be pretty expensive for most people. I'm not sure this would be a good idea, especially since there are better ways to hide modules (e.g., directly through Lmod).

This is a valid issue, but for now I would suggest you just avoid using hidden = True, especially since I suspect it will also play havoc with your use of modulerc (the redirect 11.6 will not be visible to the users I think if the target module is hidden).

[Flamefire]

This is a valid issue, but for now I would suggest you just avoid using hidden = True, especially since I suspect it will also play havoc with your use of modulerc (the redirect 11.6 will not be visible to the users I think if the target module is hidden).

Good point, thanks!