Proposal: allow all configuration parameters at any level

[sjpb]

I keep running in to this - I think it would be really useful if any of the accepted configuration parameters could be specified in system, partition, environment or test, with "deeper" settings overriding "shallower" ones. Two examples of use for this:

• The scheduler's default time limit could be be overridden for an entire reframe partition, rather than having to set it for every test when really its at least partly partition-dependent.
• I want to set a default launcher of srun for all environments in a partition, but for some MPI where this doesn't work properly I want to use mpirun in just that environment. Currently this environment has to go in a separate partition as launcher is only definable at partition-level.

I think there's a more general pattern here than just allowing those two specific cases.

It might also make the documentation easier!

[vkarak]

That would be an interesting enhancement but needs to be well thought. The first point implies to set default values for test attributes at the configuration level. This was sth we have thought about it a couple of years ago, but we haven't pursued that further. Perhaps, we can start rethinking it. The second point, I think, it's because the launcher is related to MPI environments, so it could also be part of the environment's configuration. But, I agree, there is a pattern here that could tie everything together. The risk with that is not to end up in a situation where each configuration parameter can go anywhere and create more confusion, especially if some parameters have no meaning in some contexts.

[lcebaman]

This is indeed very much needed. A test should be able to overwrite some parameters such us the launcher.

[sjpb]

I am still not sure about @vkarak's comment above:

The risk with that is not to end up in a situation where each configuration parameter can go anywhere and create more confusion, especially if some parameters have no meaning in some contexts.

To mean, saying "there's a hierarchy system -> partition -> environment -> test, and default configurations for each may be specified in any higher level, which will be overridden by configurations at the appropriate level" makes sense. But maybe I deal with Ansible's mad variable precedence too much 😆 . This is definitely one of the things which made reframe hard for my use-cases though.

[vkarak]

Sorry for getting back so late, but better late than never, I guess :-)

Implementing this suggestion would be a drastic change in some key design concepts of how partitions and environments work and how a test interacts with them. But I understand that such a feature would have helped in the past in a situation where you'd need to define a new partition or environment just to have a new launcher, because this would require you to add this new partition to your tests' valid_systems.

However, since the introduction of partition and environment features and extras, tests are completely decoupled from specific partitions and environments. Now you can (and should, actually) define only "constraints" in valid_systems and valid_prog_environs. Take this example in the new tutorial that defines valid_prog_environs as follows:

    valid_prog_environs = ['+openmp']

This tells ReFrame that this test is valid for any environment that defines the openmp feature. Environment features are defined in the environment definitions in the config as follows (and similarly for partitions):

{
    'name': 'gnu',
    'features': ['openmp', ...]
}

You can also have extras which are arbitrary key/value pairs that can be accessed from the test. Again in the same example, your environments could define an omp_flag extra that the test could use transparently from the environment like here:

    @run_before('compile')
    def prepare_build(self):
        omp_flag = self.current_environ.extras.get('omp_flag')
        self.build_system.cflags = ['-O3', omp_flag]

The exact same principles apply for partitions, too. So if your test defines valid_systems = ['+foo -bar'], your test will run on any partition that has the foo feature and not the bar: you can add a million of them without touching your test.

If you were to give a second chance to ReFrame, that would be a "must" feature to use imho, as well as variables and parameters. In 4.6 we have completely revamped the docs to reflect all these new concepts (https://reframe-hpc.readthedocs.io/en/stable/tutorial.html).

@lcebaman

This is indeed very much needed. A test should be able to overwrite some parameters such us the launcher.

You can still do that in your test: check out here.

[sjpb]

Ah interesting, thanks. I was watching your livestream at the easybuild usergroup this morning and thought this sounded useful so thank you for the clear examples above. I am expecting to do more ReFrame soon(ish) so this does look like it will help.