easybuild-easyblocks
easybuild-easyblocks copied to clipboard
Patch `ctypes` for Python installations when filtering `LD_LIBRARY_PATH`
ctypes relies on LD_LIBRARY_PATH and doesn't respect rpath linking, this is a workaround for the EasyBuild context for this setup.
Patching seems to work (edit: but not as originally intended):
alanc@~/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks(ctypes_python)$ eb --include-easyblocks-from-pr 3352 Python-3.11.5-GCCcore-13.2.0.eb --rebuild --stop patch
== Temporary log file in case of crash /tmp/eb-7eym3fl5/easybuild-m8zrpch1.log
>> download succeeded: https://github.com/easybuilders/easybuild-easyblocks/archive/develop.tar.gz
>> running command:
[started at: 2024-06-07 15:10:31]
[working dir: /tmp/eb-7eym3fl5/tmpnvx85rzn/easybuilders]
[output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-rp8160c1.log]
tar xzf /tmp/eb-7eym3fl5/tmpnvx85rzn/easybuilders/develop.tar.gz
>> command completed: exit 0, ran in < 1s
>> download succeeded: https://github.com/easybuilders/easybuild-easyblocks/pull/3352.diff
== easyblock python.py included from PR #3352
WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* EESSI-extend/2023.06-easybuild
This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.
To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.
See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.
== found valid index for /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs, so using it...
== Running parse hook for Python-3.11.5-GCCcore-13.2.0.eb...
== found valid index for /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs, so using it...
== Running parse hook for GCCcore-13.2.0.eb...
== Running parse hook for GCCcore-13.2.0.eb...
== processing EasyBuild easyconfig
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/generic/software/EasyBuild/4.9.1/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb
== building and installing Python/3.11.5-GCCcore-13.2.0...
>> installation prefix: /home/alanc/eessi/versions/2023.06/software/linux/x86_64/generic/software/Python/3.11.5-GCCcore-13.2.0
== fetching files...
>> sources:
>> /tmp/alanc/easybuild/sources/p/Python/Python-3.11.5.tgz [SHA256: a12a0a013a30b846c786c010f2c19dd36b7298d888f7c4bd1581d90ce18b5e58]
>> download succeeded: https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=59e2ce0e6b46bb47ef81b68b600ed087e14fdaad
== ... (took 1 secs)
== creating build dir, resetting environment...
>> build dir: /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0
== Running post-ready hook...
== ... (took < 1 sec)
== unpacking...
>> running command:
[started at: 2024-06-07 15:10:36]
[working dir: /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0]
[output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-9o1mqhq7.log]
tar xzf /tmp/alanc/easybuild/sources/p/Python/Python-3.11.5.tgz
>> command completed: exit 0, ran in < 1s
== ... (took < 1 sec)
== patching...
== ... (took < 1 sec)
>> running command:
[started at: 2024-06-07 15:10:37]
[working dir: /home/alanc/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks]
[output logged in /tmp/eb-7eym3fl5/easybuild-run_cmd-i09p25cp.log]
bzip2 /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/easybuild/easybuild-Python-3.11.5-20240607.151037.log
>> command completed: exit 0, ran in < 1s
== COMPLETED: Installation STOPPED successfully (took 2 secs)
== Results of the build can be found in the log file(s) /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/easybuild/easybuild-Python-3.11.5-20240607.151037.log.bz2
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-7eym3fl5/easybuild-m8zrpch1.log* have been removed.
== Temporary directory /tmp/eb-7eym3fl5 has been removed.
alanc@~/EasyBuild_Git/easybuild-easyblocks/easybuild/easyblocks(ctypes_python)$ grep LD_L /tmp/alanc/easybuild/build/Python/3.11.5/GCCcore-13.2.0/Python-3.11.5/Lib/ctypes/util.py
libpath = ':'.join(filter(None, os.environ.get('LD_LIBRARY_PATH', '').split(':') + os.environ.get('LIBRARY_PATH', '').split(':')))
@boegelbot please test @ jsc-zen3 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"
@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de
PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3352 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!
- exit code: 0
- output:
Submitted batch job 4318
Test results coming soon (I hope)...
- notification for comment with ID 2154930449 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
@boegelbot please test @ generoso EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"
@ocaisa: Request for testing this PR well received on login1
PR test command 'EB_PR=3352 EB_ARGS="--trace Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!
- exit code: 0
- output:
Submitted batch job 13671
Test results coming soon (I hope)...
- notification for comment with ID 2154994489 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
Test report by @boegelbot
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
- SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
- SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb
Build succeeded for 3 out of 3 (3 easyconfigs in total) jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18 See https://gist.github.com/boegelbot/3cad8540c8ccb38a784472ca19e53e94 for a full test report.
@boegelbot please test @ jsc-zen3 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"
@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de
PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3352 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!
- exit code: 0
- output:
Submitted batch job 4321
Test results coming soon (I hope)...
- notification for comment with ID 2155065309 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
Test report by @boegelbot
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
- SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
- SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb
Build succeeded for 3 out of 3 (3 easyconfigs in total) cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8 See https://gist.github.com/boegelbot/809da3be74a15531dbf6c7286c67ef06 for a full test report.
@boegelbot please test @ generoso EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb"
@ocaisa: Request for testing this PR well received on login1
PR test command 'EB_PR=3352 EB_ARGS="--trace Python-3.11.3-GCCcore-12.3.0.eb Python-3.11.5-GCCcore-13.2.0.eb Python-3.10.4-GCCcore-11.3.0-bare.eb Python-3.10.8-GCCcore-12.2.0-bare.eb Python-3.11.2-GCCcore-12.2.0-bare.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3352 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!
- exit code: 0
- output:
Submitted batch job 13673
Test results coming soon (I hope)...
- notification for comment with ID 2155132179 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
Test report by @boegelbot
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
- SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
- SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
- SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
- SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb
Build succeeded for 5 out of 5 (5 easyconfigs in total) jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18 See https://gist.github.com/boegelbot/bbb52ea4e37e830361b159c1bb849f5c for a full test report.
Test report by @boegelbot
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
- SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
- SUCCESS Python-3.10.4-GCCcore-11.3.0-bare.eb
- SUCCESS Python-3.10.8-GCCcore-12.2.0-bare.eb
- SUCCESS Python-3.11.2-GCCcore-12.2.0-bare.eb
Build succeeded for 5 out of 5 (5 easyconfigs in total) cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8 See https://gist.github.com/boegelbot/586ae36ed6c653da99c4548e9da3cd6b for a full test report.
Ok, the latest commit fixes the problem:
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module load foss
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module load Python
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ module unload EESSI-extend/2023.06-easybuild
Due to MODULEPATH changes, the following have been reloaded:
1) Python/3.11.5-GCCcore-13.2.0
{EESSI 2023.06} ocaisa@LAPTOP-O6HF2IKC:~$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libmpi.so.40: cannot open shared object file: No such file or directory
However, I'm still seeing if there is a better way
@mboisson You may have some insight for this one and want to comment?
Seems to work for me
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ module load Python/3.11.5-GCCcore-13.2.0
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ which python
~/eessi/versions/2023.06/software/linux/aarch64/neoverse_n1/software/Python/3.11.5-GCCcore-13.2.0/bin/python
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ module load foss
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ module unload EESSI-extend
Due to MODULEPATH changes, the following have been reloaded:
1) Python/3.11.5-GCCcore-13.2.0
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ which python
/cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/neoverse_n1/software/Python/3.11.5-GCCcore-13.2.0/bin/python
{EESSI 2023.06} [ocaisa@aarch64-neoverse-n1-node1 ~]$ python -c "from ctypes.util import find_library; from ctypes import CDLL; CDLL(find_library('mpi'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/neoverse_n1/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libmpi.so.40: cannot open shared object file: No such file or directory
The first part (returning the full path) seems to be similar to what we do in https://github.com/ComputeCanada/custom_ctypes
but I don't recall changing ldconfig path.
We opted to make it in a separate module which shadows Python's built-in ctypes just in case it caused trouble. We only load this module (and therefore shadow Python's built-in ctypes) in very rare circumstances.
For what it's worth, this is the diff for python 2.7 to 3.9: https://github.com/ComputeCanada/custom_ctypes/commit/b3a0288f6e6174b502107a92ac8cd0db0842411b
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.10.4-GCCcore-11.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/5a508de44555f1d9590331cff1411c8d for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.10.8-GCCcore-12.2.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/5cc6e3cd5f45d7477e883e3cfc41e0ef for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/1b2dd477cc103b7c77ba6f0553d452d7 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/0e16cb9edf85e76fcbff10f0839b0681 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Biopython-1.79-foss-2022a.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/9f8ed62e326fb88e9d09c1bce1dfe708 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Biopython-1.81-foss-2022b.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/ad349eba48f7e55ef3ee2c0fe9e129cc for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Boost.Python-1.79.0-GCC-11.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/a20eda766fe31634a4052bf5a25d5d56 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Boost.Python-1.83.0-GCC-13.2.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/c3fda5271d5d8f359f6496e71903dd57 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS Brotli-python-1.0.9-GCCcore-11.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/00592dd2f8f735cce1fb80907df571c3 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS flatbuffers-python-23.5.26-GCCcore-12.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/92791ce25a77e56870ec8b441dcbf135 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS GitPython-3.1.31-GCCcore-12.2.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/1d98b369e266650d8499db3e41a6cde2 for a full test report.
Test report by @casparvl
Overview of tested easyconfigs (in order)
- SUCCESS GitPython-3.1.40-GCCcore-12.3.0.eb
Build succeeded for 1 out of 1 (1 easyconfigs in total) tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8 See https://gist.github.com/casparvl/22de6d0fb8db85c02a31c925fd6a2a14 for a full test report.