software-layer
software-layer copied to clipboard
Enable partial builds of software layer stack
The current EESSI-pilot-install-software.sh works in two main steps:
- Building (missing) software packages.
- Creating a single tarball for new files added through step 1.
This is challenging for larger software stacks where some package does not build successfully, maybe just for temporary reasons (e.g., inability to download a source file).
If the building and tarball creation could be interleaved, it would allow a much improved incremental building of software stacks. For every EasyBuild package being built and installed (including dependencies), a tarball could be created. Individual tarballs could be ingested which would speed up the build procedure because packages are already available.
If ingesting too many small tarballs turns out to be problematic, individual tarballs could be joined before they are ingested.