`multi_deps` with a binary + extension package does not work

[mboisson]

trafficstars

I am trying to install TensorRT, which supports multiple versions of python. I want to use multi_deps

The problem is that the tensorrt.py EasyBlock is based on a binary. Extracting the binary multiple times simply over-writes the existing installation folder for each version of python. The resulting installation therefore only supports the last version of python specified in the multi_deps

https://github.com/ComputeCanada/easybuild-easyblocks/blob/computecanada-master/easybuild/easyblocks/t/tensorrt.py

https://github.com/ComputeCanada/easybuild-easyconfigs/blob/computecanada-master/easybuild/easyconfigs/t/TensorRT/TensorRT-7.0.0.11-CUDAcore-10.1.243.eb

@bartoldeman @boegel, any idea how to work around this ?

It seems like for a Binary EasyBlock, multi_deps should perform the installation step only once.

[mboisson]

I tried this to see if it would keep the files between each run keeppreviousinstall = True

But it does not seem to do the trick :/

[mboisson]

I see @akesandgren was the author of the tensorrt.py EasyBlock, so tagging you too.

[mboisson]

The issue is caused by this line : https://github.com/ComputeCanada/easybuild-easyblocks/blob/computecanada-master/easybuild/easyblocks/generic/binary.py#L109

which can only be removed with Python 3.8's new dirs_exist_ok for the shutil.copytree

This means that the option keeppreviousinstall = True

is not compatible with packages that use the Binary EasyBlock.

In general, this also means that any EasyBlock that inherits from Binary can not successfully use the multi_deps option, without a workaround (see next post).

[mboisson]

For my specific use case (TensorRT), I worked around the issue by bypassing this codepath by specifying a custom install_cmd :

https://github.com/ComputeCanada/easybuild-easyconfigs/blob/6b883300c207014bf1fe7739f749dd9906401fec/easybuild/easyconfigs/t/TensorRT/TensorRT-6.0.1.5-CUDAcore-10.1.243.eb#L15

[damianam]

We have been hit by this as well, while preparing an easyconfig file for NVPL, based on the Bundle easyblock and components, with the PackedBinary as default easyblock.

The workaround with a custom install_cmd works, even though then there is an additional hiccup (not showstopper). The hiccup is that for every component the PackedBinary easyblock iterates over the list of sources and calls then the install method of the Binary easyblock, meaning that the install_cmd is executed N*N times, instead of just N times.