easybuild-easyblocks
easybuild-easyblocks copied to clipboard
PoC: update `PythonPackage` easyblock to build wheel first when installing Python package with `pip`
(created using eb --new-pr)
This is a Proof of Concept only.
It adds the option to set use_pip = 'wheel' which matches more our build-test-install cycle.
Currently when using pip we have this:
- build step does nothing
- test step installs into a temporary directory with
pip install --prefix /tmp/foo . - install step installs to the final location with
pip install --prefix /final/foo .
Confusingly all build options need to be passed to (pre)installopts and as Python usually creates a "wheel file" (basically a compiled, installable zip) this duplicates work.
This option when set does the following:
- build step:
pip wheel --wheel-dir /new-tmp-dir - test step installs into a temporary directory with
pip install --prefix /tmp/foo /new-tmp-dir/*.whl - install step installs to the final location with
pip install --prefix /final/foo /new-tmp-dir/*.whl
This avoids duplicate work and ensures that what is tested is what gets installed (currently we build twice which could yield different results). It is also nice to have separate timing and output for the actual build
However it turned out a bit more involved than anticipated. E.g. not all options for pip install apply to pip wheel and vice versa. Also the time savings are not that great. I tested PyTorch 2.1.2 on a (admittedly high-cpu) machine. And after an initial build/install the times are:
setup.py install: ~35spip install .: ~63spip install *.whl: ~17s
So the time saving is around 11s which IMO isn't worth the added complexity.
However it brings back a dedicated build step, which shows up as such in EB status/progress and not confusingly in the test or install step.