software-layer icon indicating copy to clipboard operation
software-layer copied to clipboard

distribute building of software layer across nodes in a cluster

Open boegel opened this issue 4 years ago • 1 comments

We're currently always using a single node to build the software layer, it would be nice to have support for leveraging a pool of available nodes, which should significantly speed up the process.

TODOs:

  • [ ] fully autonomous installation procedure using EESSI build container (incl. running in Gentoo Prefix environment)
  • [ ] set up overlay directories in a shared filesystem (?)
    • or use local filesystem and create tarballs per installation, which are expanded as-needed through an EasyBuild hook
  • [ ] modify existing of add separate build script
  • [ ] add support to EasyBuild to provide a template for a job script to allow building in a container image

boegel avatar Nov 15 '21 13:11 boegel

Have you looked at using distcc? I never tried it on prefix, I wonder if that would work at least for the compatibility layer.

amadio avatar Nov 16 '21 07:11 amadio