Module Dependencies: moduledeps for shpc

[georgiastuart]

Is your feature request related to a problem? Please describe.

Singularity on HPC has a well known limitation where containers built with the hybrid model need compatible MPI stacks inside the container and outside the container for multi-node parallelism. On our HPC systems we have a variety of MPI stacks that differ based on implementation (MPICH, MVAPICH, OpenMPI, Intel), CUDA capability, etc. Right now, I have shpc installed on a relatively simple GPU cluster with only OpenMPI+CUDA. My concern is when we roll out shpc to our large CPU cluster that has a hodgepodge of MPI stacks then users will load an incompatible container.

Describe the solution you'd like I would like to be able to arrange containers in an lmod moduledeps hierarchy. So the modules folder could look like:

modules
| openmpi4
| | mpi-dependent-container
| mpich
| | mpi-dependent-container
| non-mpi-dependent-container-1
| non-mpi-dependent-container-2

Then, I'd edit the openmpi4 and mpich module files to append the modules/<mpi stack> directories to $MODULEPATH (this could not be governed by SHPC).

The solution I've thought of is as follows:

In the container.yml registry file, there could be a line like

moduledeps: openmpi4

with a corresponding entry in the shpc config file so that it wouldn't try to load openmpi4 as a module:

moduledeps:
  - openmpi4
  - mpich

I could also see support for nested module deps, like with a compiler dependency. container.yml:

moduledeps: gnu8/openmpi4

And the shpc conf:

moduledeps:
  gnu8:
    - openmpi4
    - mpich
  intel:
    - openmpi4
    - impi

Describe alternatives you've considered Adding module swapping/loading to the wrapper scripts, but I would prefer not to change the user's module environment.

Not sure if this is feasible or not! I'm happy to take a crack at it once we get the details hammered out.

[georgiastuart]

Alternative directory hierarchy:

modules
| non-mpi-dependent-container-1
| non-mpi-dependent-container-2
moduledeps
| openmpi4
| | mpi-dependent-container
| mpich
| | mpi-dependent-container

[vsoch]

Oh I really like this idea! So the only problematic thing would be requiring some container module to load other modules - especially for recipes in the main registry (likely not your personal/custom ones) this could be problematic given the basic reality that module names differ across clusters. But I could see this working for either 1. your custom or private container recipes, or 2. if it's an optional install and has module names that are common across centers. Specifically this recipe:

moduledeps:
  gnu8:
    - openmpi4
    - mpich
  intel:
    - openmpi4
    - impi

could work, because the default install would not use them:

shpc install tensorflow/tensorflow

But if the user could ask for it, we'd grab the module set they specify and add to the install directory:

shpc install tensorflow/tensorflow +gnu8

or it could even be more "on the fly"

shpc install tensorflow/tensorflow +openmpi +mpich

And then the issue, of course (as you pointed out) becomes the install tree. We couldn't have two of the same container (with different depends on) under the same tensorflow/tensorflow.

Given that we are allowed to install containers with a different name (albeit predictable so we can uninstall) let's think through the directory structure. For this one:

modules
| openmpi4
| | mpi-dependent-container
| mpich
| | mpi-dependent-container
| non-mpi-dependent-container-1
| non-mpi-dependent-container-2

I like that the containers are separate from the module dependencies, but we'd get a namespace conflict, say, if we tried installing a container that falls under the tag for the module dependencies install (let's call this a variant, to take a term from spack). E.g.,:

modules
| openmpi4
| | 1.0.0  <--- version from container
| | mpi-dependent-container <- now at the wrong level, ohno!
| mpich
| | mpi-dependent-container
| non-mpi-dependent-container-1
| non-mpi-dependent-container-2

And if I'm browsing top level folders, it would be confusing to know what is module variant vs. container URI. For this idea:

modules
| non-mpi-dependent-container-1
| non-mpi-dependent-container-2
moduledeps
| openmpi4
| | mpi-dependent-container
| mpich
| | mpi-dependent-container

This makes more sense to not mix up the two, but then if we have a container that is installed with and without a module variant we get:

modules
| container-1
moduledeps
| openmpi4
| | container-1

and it becomes harder to know that there are two, just one variant. And uninstall happens in two places. This isn't an impossible way to do it, but I don't think it's quite right for those reasons!

But actually the idea of having nested module directories isn't so bad - e.g., you can install to (and then load) modules that are intended for specific purposes. For the example above:

modules
    gnu8
        tensorflow/tensorflow
    intel
        tensorflow/tensorflow    

And then to install, it could be either on the fly (you decide the variants and directory)

shpc install tensorflow/tensorflow +openmpi4 +mpich --dest gnu8
shpc install tensorflow/tensorflow +openmpi4 +impi --dest intel

That would be infinitely flexible to the modules structure and names for grouping of dependencies - and we'd have to be careful with uninstall commands, e.g., they could parse over modules and every subdirectory (variant) to look for a particular module, OR we could adapt to force the user to choose a specific destination directory. This would also be messy if we are reorganizing the module install directory entirely, so we should think about it first. For commonly associated dependencies, we might just go ahead and put them in the container.yaml as you showed:

moduledeps:
  gnu8:
    - openmpi4
    - mpich
  intel:
    - openmpi4
    - impi

And then install could look the same, but instead have the + (or -v for variant) be the variant name:

shpc install tensorflow/tensorflow +gnu8
shpc install tensorflow/tensorflow +intel

And then shpc would manage that namespace, each of gnu8 and intel.

Hmm, so I love this idea but I need to think about it more. Essentially we want to be able to specify dependencies, sometimes even custom dependencies, but still be able to allow shpc to manage the space. And in module speak a dependency is another module name, with or without a version, and within or outside of shpc (the harder use case!) If we have common names for variants across container.yaml files (e.g., intel) that's also making a declaration that a specific set of modules should be grouped together (under the same module use ./path potentially) so we should consider that too.

ping @marcodelapierre I think you will like this idea too! I'm going to munch on it a bit. I think minimally we want to finish and merge #517 first so when we work on this we can be sure the default version functionality is maintained / ok.

[vsoch]

Okay more brainstorming. I think we need to be careful to preserve backwards compatibility. So my thinking is we should structure the install to have variants like:

└── vanessa
    ├── pokemon
    │   └── latest
    │       ├── variant-a/
    │       │       ├── 99-shpc.sh
    │       │       └── module.lua
    │       ├── 99-shpc.sh
    │       └── module.lua

The idea above is that variant-a is a named variant of my choosing, and it is scoped to be inside of a particular version. The module software (I hope) would then let me load:

module load vanessa/salad/latest/variant-a

Under this setup we define a variant, which is some customization of a container.yaml. This means that we can technically over-ride any attributes for the container, or for our purposes right now, we just need moduledeps. So variants can be named thing, and always optional for an install. Variants provided by shpc would look like this:

registry/vanessa/
    pokemon
        container.yaml
        variants/
             variant-1.yaml

And maybe the variant file would have:

moduledeps:
  - openmpi4
  - mpich

On a high level, it's allowing a user to define and install any number of customized primary container recipes. Then to install

shpc install vanessa/pokemon +variant-1

And that would generate the install tree shown above. And maybe for "on the fly" variants you would do:

shpc install vanessa/pokemon +variant-1 --variant-file variant-1.yaml

And that says "use variant-1.yaml to define a custom variant for vanessa/pokemon named "variant-1' (the +variant-1 bit). We force people to write them into files for a tiny bit of reproducibility I think.

All of this depends on the ability for this extra nesting to work - haven't tried it before!

docker: tensorflow/tensorflow url: https://hub.docker.com/r/tensorflow/tensorflow maintainer: '@vsoch' description: An end-to-end open source platform for machine learning. latest: 2.7.1-gpu: sha256:581575fc3a736398f0dff9e950f57f2e6d808296267ac98325451a0b1d101dd0 tags: 2.5.0-custom-op-gpu-ubuntu16: sha256:478bee6f0691b48d74adc3fcffe3e9ececf35df5c02860cc51a2c48b1d92c730 2.5.0rc0-gpu-jupyter: sha256:9808e04142b09482bb6b3d1738430ae7472a214dd38e086d41e481b376fa9abd 2.6.0: sha256:773d5ce09e4ce003db02740c6a372a8a9f43be2bac23544d8f452bfec5347c53 2.6.0rc0-gpu-jupyter: sha256:358b5bf90aaf4e56813ff22f2981d86fab7ddc59552b0be6022ae04d6a9f43c3 2.7.0: sha256:31e09cf438a41f12c759cc8cc79c6b0fbb0db5abfc3de8169e916c8c9ac38dc5 2.7.0rc0: sha256:abbc457c9b7c0725d7d0db885dbb313db3d0ae25733b083900a508efb672af94 2.8.0: sha256:7c01f75d58fadc2cd1109d5baac1925ed131e05925d840b1b49363c794d1c4db 2.8.0rc0: sha256:11e5d21a786da523d2f7de530c083d5c72a06e02c8895c84595d107c579027a1 latest-gpu: sha256:1e03623e335aac1610b1a3cfa6a96cf10156acb095287f9d6031df3980148663 2.7.1-gpu: sha256:581575fc3a736398f0dff9e950f57f2e6d808296267ac98325451a0b1d101dd0 filter:

• 2.*
• latest-gpu features: gpu: true aliases: python: /usr/local/bin/python

We could then 

[georgiastuart]

I love the variant idea! This solves a few issues:

1. I was puzzling over how I wanted to jam the new $MODULEPATH path into the MPI stack module files in a way that could go into our automation system. This means there's no need to append to the modulepath since all variants would be visible to the user at all times and the responsibility would be on the user to pick the right one. But it will hopefully be clear to the user which one is correct.
2. This also lets us have +gpu variants and +cpu variants of containers, or any other arbitrary variant. This would really help on our condo system where we have a mix of nodes with GPUs and nodes without.
3. I love the idea of having a variant.yml file that overrides container.yml. I could see it being used to provide a variety of wrappers, for instance, with different use cases for the same container, etc. However, the base container url/sha needs to also be overwritable because in the case of different MPI stacks or GPU vs CPU then they'll likely be completely distinct containers.

Open Questions

What if a user/cluster manager wants more than one variant?

Should it be allowed to specify arbitrary numbers of variants? For a complicated example:

└── vanessa
    ├── pokemon
    │   └── latest
    │       ├── variant-a/
    │       │       ├── 99-shpc.sh
    │       │       └── module.lua
    │       ├── variant-b/
    │       │       ├── 99-shpc.sh
    │       │       ├── module.lua
    │       │       ├── nested-variant/
    │       │       │       ├── 99-shpc.sh
    │       │       │       └── module.lua
    │       ├── 99-shpc.sh
    │       └── module.lua

The above has pokemon:latest installed in 4 ways:

shpc install pokemon:latest
shpc install pokemon:latest +variant-a
shpc install pokemon:latest +variant-b
shpc install pokemon:latest +variant-b +nested-variant

The use case I could see for this is something like

shpc install pokemon:latest +openmpi +cuda
shpc install pokemon:latest +openmpi +rocm

i.e., when the same container with an openmpi stack is built for either NVIDIA or AMD GPUs as distinct containers with distinct variant.yml files.

What if a user tries to use an inappropriate container anyway?

My instinct is this is a a problem best solved on a container-by-container basis with customizations for the system (using the newly revamped add function perhaps!). As you pointed out, module names vary widely between systems.

This is probably the topic for another issue, but a way to jam a prereq into the modulefile would solve this. It does not look like it's currently possible to add to the modulefile, but I think something similar can be accomplished by overrriding the default wrapper script for singularity to load prereq modules. E.g.,

container.yml or variant.yml

prereq:
  - cuda11
  - openmpi4

prereq-module-snippet.sh

{% if config.prereq %}
{% for prereq in config.prereq %}
if ! module is-loaded {{ prereq }}; then echo "{{ prereq }} is not loaded";   exit 1; fi
{% endfor  %}
{% endif %}

singularity.sh

...
{% include 'prereq-module-snippet.sh %}
...

Well that went a bit on a tangent!

[marcodelapierre]

This sounds very interesting, and MPI + containers is one of my, ehm, little obsessions :-D I will aim to give this thread a good read over these couple of days, to see if I can provide any useful perspective.

[marcodelapierre]

Hi folks,

sorry, it took me a bit to get back to this thread. I read most of it, and wow! it was a worthwhile reading on a very interesting topic!
As a result, I have a salad of comments in my mind, so I will do my best to share some of them here in a clear, not confusing, way :-)

The first general comment is that when we talk MPI and modules, various computing centres can take quite different approaches, each of which can have different implications and dependencies. So, I will try to comment on a very narrow topic here, to try and see if we can envisage a solution that can be flexible, without being too complex. (I appreciate that complexity may need to be put back in later)

I will then avoid talking of variants in general (eg regarding different compilers and so on), and focus only on the MPI aspect instead. In my experience, which has inevitably some limits, I have luckily never needed so far to juggle with weird conflicts between host/container libraries, stuff like LIBC and similar. As a result, the only two cases where I have seen the need for multiple host library bindings are GPU (which Singularity handles nicely, thanks of standardisation of library paths kindly provided by Nvidia and AMD) and, indeed, MPI.

Sorry, I am going to make the story a bit long here, hoping that it can help in finding a simple and cross-centre flexible solution to our MPI struggles.

I see MPI as a bad beast. There are different MPI implementations, and also different underlying sets of interconnect libraries, for which each cluster has different settings: often modules for MPI, and typically arbitrary path and filenames for interconnect ... a nightmare!

What I learnt is that, although there are many MPI implementations, when it comes to ABI compatibility of the library there are only two families: what I call the MPICH family (MPICH, MVAPICH, Intel, Cray and others, see https://www.mpich.org/abi/) and OpenMPI, all alone. At our HPC centre, among others we offer support for MPI containers. Thanks to the ABI compatibility above, our researchers need at most 2 different containers for each application: an MPICH one, and an OpenMPI one. This literally means that for each MPI application we need to build and deploy at most only two container "variants".

The other bit we need is two distinct sets of Singularity environment variables, one for MPICH containers and one for OpenMPI containers, that set the appropriate bind mounting paths and LD_LIBRARY_PATH definitions for the two cases. The specific way we implement this is by having 2 Singularity modules, a "singularity" one (MPICH), and a "singularity-openmpi" one. Other implementations are certainly legitimate though. One notable feature of our implementation is that we don't need to load mpich/openmpi modules themselves, the few relevant bits are embedded in the two singularity modules. Once we have these two modules setup, any MPI container can run on our infrastructure.

Now, in our approach, the containers that will run using either the host "MPICH-ABI-compatible" or "OpenMPI" libraries will need to have been built either with MPICH or OpenMPI. In other words, we have two Dockerfiles. And notably, because this MPI ABI distinction is so crucial for MPI applications, we made the choice to enforce a naming convention where the "mpich" and "openmpi" variants have distinct container repository names, eg "salad-mpich", and "salad-openmpi". This implies that in SHPC these two variants will have two distinct recipes, which makes thing easier I think. Side note: we also support CUDA+MPI containers; in our approach, we have no need to load the Cuda module from the host, the only thing that is needed in the container is bind mounting the host driver for the GPU card, which is unique for each compute node, and which is typically automatically identified by Singularity ("gpu" feature in SHPC).

So, my first proposal. First, consider adopting a naming convention for MPI containers, that explicitly states the underlying ABI compatibility, "mpich" or "openmpi", eg in the form of a suffix. This will help users. Then, we could implement a new "mpi" feature in the SHPC recipe, similar to "gpu" and so on, that will say if the container is built with "mpich" or "openmpi" libraries. (default for "mpi" would be otherwise "false", same as for "gpu"). Then back in the main settings of SHPC, we could implement the site configuration for MPI, where the key feature is probably just a set of extra modules to load. To make it flexible for various centres, I would probably think of an implementation that avoids conflicts with the container runtime modules. For instance, we could implement and document that, for MPI containers, singularity_module and similar are ignored, and all module loads are taken from the MPI setting instead. With this setup users, that type module avail for SHPC modules, will see salad-mpich and salad-openmpi, and all remains to be done is to document when to use which.

I apologise for the long message, and for the inevitable simplifications I have made. @georgiastuart , I specifically apologise to you, because high chances are that my simplifications have made my message divert from your original needs. As I said, I appreciate there are several approaches to deploy MPI stuff, so I can only hope that you can find at least a tiny bit of what I wrote useful. But maybe I was really completely out of scope.

Also, I am sure there are precious insights I could learn by listening to your experience in deploying and exposing to researchers MPI frameworks and applications. So, please take my horribly long message as "I would be super keen to further the conversation and find optimal ways to deploy MPI containers, using SHPC".

If you got down to this point, thanks for your patience in reading! Talk later, Marco

[marcodelapierre]

And just to bring some code for my suggestion.

1. [ Guideline only, no implementation] Naming convention for recipe entries to include MPI flavour, eg: vanessa/salad-mpich and vanessa/salad-openmpi

2. New feature in the container recipe:

features:
  ..
  mpi:   # can be "mpich", "openmpi" or "false"/"null" (latter is default)

3. New feature in the SHPC settings:

mpi_mpich:   # null (default), or list of modules to load (overrides singularity_module/podman_module)
mpi_openmpi:   # null (default), or list of modules to load (overrides singularity_module/podman_module)

[vsoch]

Oh this is super valuable! So I don't use much at our center (to be honest I'm not a huge fan of HPC, ironic that I do so much software engineering for it) so I don't have this perspective. Specifically this idea:

The other bit we need is two distinct sets of Singularity environment variables, one for MPICH containers and one for OpenMPI containers, that set the appropriate bind mounting paths and LD_LIBRARY_PATH definitions for the two cases. The specific way we implement this is by having 2 Singularity modules, a "singularity" one (MPICH), and a "singularity-openmpi" one. Other implementations are certainly legitimate though. One notable feature of our implementation is that we don't need to load mpich/openmpi modules themselves, the few relevant bits are embedded in the two singularity modules.

and then embedding the mpi flavor in the container name I really like. It's pretty do-able to provide different recipes for different mpi flavors - it's just a yaml file! That seems much easier than say, adding this complex variants thing.

Let's see what @georgiastuart thinks! it could be what we just need to do is have documentation or instructions / template for how to roll a new flavor. And then some limited set of features to turn things on and off.

Also - I likely am going to be implementing the updates to happen from shpc directly - I think binoc is going to be deprecated and we can't rely on it moving forward. I'll be happy to bring back if it comes back, but probably this weekend I'm going to add the update functionality here instead.

[vsoch]

@marcodelapierre I love that!! It's seriously so simple, I feel really silly for not thinking of it.

[marcodelapierre]

Just some implementation notes that came to mind. We can have a similar mechanics than the GPU feature. So, SHPC will look at the container recipe:

• if mpi == false, do as usual
• if mpi == mpich, use mpi_mpich global setting, ....ignore if latter is null(?)
• if mpi == openmpi, use mpi_openmpi global setting, ....ignore if latter is null(?)

And, I agree, as you suggest, some good documentation and guidelines on the topic.

Stemming from this stimulating conversation, I am going to open a ticket to possibly update the GPU feature in a similar way.

[marcodelapierre]

Yes, I think feedback from @georgiastuart , and possibly other centres (?) would be crucial, because MPI can be deployed/configured in quite different ways. Mapping these various ways in modules to be loaded is coherent with SHPC origins (stick together a container runtime with a module system), and might do for many, but it's super important to put this hypothesis under test.

[muffato]

The specific way we implement this is by having 2 Singularity modules, a "singularity" one (MPICH), and a "singularity-openmpi" one.

Hi @marcodelapierre . I'm curious about something. Have you considered having a module tree for MPICH-enabled containers, and another one for openmpi-enabled containers ? Then module load singularity and module load singularity-openmpi, on top of adding singularity to $PATH, also modify $MODULEPATH to the right module tree ? Or is it what you want to move away from ?

[marcodelapierre]

Thanks @muffato for bringing in this extra perspective! To be honest, we're kind of lucky in this space: our main supercomputer has only Cray MPI; in the other clusters we do have both Intel MPI and Open MPI, but only support the latter for users. As a result, we're good to go with just a single container tree. For users that need one specific MPI flavour (eg because it's the one that ships in a 3rd party container), they will manage it themselves, the naming convention is just a suggestion for them.

But your point is actually good, we could maybe have extra functionality for the mpi feature, which might turn handy for @georgiastuart as well. Something like:

mpi:
  mpich:   # null (default), or list of modules to load (overrides singularity_module/podman_module)
  openmpi:   # null (default), or list of modules to load (overrides singularity_module/podman_module)
  hierarchy:  # [..]

Here, the new hierarchy property would allow to decide whether SHPC creates the mpi modules with extra directory tree or not.

I see at least a couple of options here for hierarchy allowed values:

1. Minimal: true (creates mpi containers either under mpich or openmpi subdirectory) or false
2. More flexible: '{{ mpi flavour }}/{{ mpi version}}', or '{{ mpi flavour }}', or false . This might be handy if used in conjunction with the Lmod "hierarchies" features, which I think expects the version specification, too. In this case, we need to allow the settings to specify these version, too, which right now is missing in my snippet above.

Thoughts?

[vsoch]

I’d like to see these ideas refactored in the space of “symlink trees” or shpc environments - e.g extending the symlink tree idea to be about installing named groups of modules, for which the user could perhaps specify what they want for some flavor of mpi in the equivalent of an environment file, that generates the tree, and they are loaded together.

[marcodelapierre]

good point, I will think about the environment concept a bit more!

[georgiastuart]

Wow I missed a lot over the past few days! I'm going to read through everything carefully and add my thoughts ASAP.

[georgiastuart]

• if mpi == false, do as usual
• if mpi == mpich, use mpi_mpich global setting, ....ignore if latter is null(?)
• if mpi == openmpi, use mpi_openmpi global setting, ....ignore if latter is null(?)

I am way late to respond to this, but I really like this idea. Not sure how it'll work if it requires loading a certain compiler though? Maybe extend to

• if compiler == false, don't purge modules
• if compiler == intel, purge modules and module load intel
• if compiler == gnuX, purge modules and module load gnuX

(and then do the MPI load)

Or something like that.

[vsoch]

I think compiler would have to be a separate thing? And this seems like a good use case for a view - where you could define the set of modules that you want to be loaded together. Thoughts @marcodelapierre ?

[georgiastuart]

Maybe using lmod's depends_on (or prereq) would work?

In the config

depends_on:
    - gnu9
    - openmpi

Then in the modulefile

depends_on("gnu9", "openmpi")

Then at least it'll prompt the user to load the correct mpi stack if it's not already and avoid shpc having to be opinionated.

[vsoch]

Yeah I like that! I'll make some time this weekend to draft up a PR.

[vsoch]

hey @marcodelapierre @georgiastuart @muffato - just started on this today! So I realized that we don't have full support for system modules (and thus depends_on) in views, so we are basically missing a strategy using module software to "run these extra commands when you load the module). E.g., right now with the symlink, we are just doing that (and can't write new content to the file).

For a refresher on views, the current structure of the view is to have the view.yaml with metadata, and then the symlinked module next to it, and a .version file that controls loading versions (akin to our actual install directories):

views/
├── mpi
│   ├── python
│   │   └── 3.11-rc.lua -> /home/vanessa/Desktop/Code/shpc/modules/python/3.11-rc/module.lua
│   └── view.yaml
└── second-mpi
    ├── emacs
    │   └── 27.2.lua -> /home/vanessa/Desktop/Code/shpc/modules/ghcr.io/autamus/emacs/27.2/module.lua
    └── view.yaml

So adding depends_on, that would mean we store this metadata with the view:

view:
  name: mpi
  modules:
  - python:3.11-rc
  system_modules: []
  depends_on: []

But we don't currently do anything with depends_on and system_modules. If we want LMOD and environment modules to have support for these extra bits, where do we put that logic? :thinking: Is there a supplementary file that we can add that will reliably be loaded, akin to .version? Is it just a matter of adding "more stuff" to the .version file (and always writing it in the cases we don't).

[vsoch]

Lol I think I spoke too soon - I forgot about this guy!

$ shpc view add mpi depends_on thing
Wrote updated .view_module: /home/vanessa/Desktop/Code/shpc/views/mpi/.view_module

the .view_module! I think we figured out that can handle extra logic? Let me try using that (sorry been a while!)

It's always astounding when me of the present is a complete idiot, and me of the past just had this figured out! :laughing:

[vsoch]

okey doke! Please see https://github.com/singularityhub/singularity-hpc/pull/591

[vsoch]

OK we've added depends_on in https://github.com/singularityhub/singularity-hpc/pull/591, and that with support for system modules should satisfy this issue. There was some other discussion in this issue about mpi variables, and @marcodelapierre I apologize if I've fallen a bit behind here - please ping me on other issues you want me to work on next/soon!

[georgiastuart]

Awesome, thanks so much @vsoch !