easybuild-easyconfigs
easybuild-easyconfigs copied to clipboard
easybuilders
{bio}[GCC/11.2.0] Raven v1.8.1 w/ Python 3.9.6
@ItIsI-Orient: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-easyconfigs/actions/runs/2636690868 Output from first failing test suite run:
FAIL: test_pr_python_packages (test.easyconfigs.easyconfigs.EasyConfigTest)
Several checks for easyconfigs that install (bundles of) Python packages.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/easyconfigs/easyconfigs.py", line 1035, in test_pr_python_packages
self.fail('\n'.join(failing_checks))
AssertionError: 'download_dep_fail' should be set in exts_default_options in Raven-1.8.1-GCC-11.2.0.eb
'use_pip' should be set in exts_default_options in Raven-1.8.1-GCC-11.2.0.eb
----------------------------------------------------------------------
Ran 15037 tests in 619.037s
FAILED (failures=1)
ERROR: Not all tests were successful
bleep, bloop, I'm just a bot (boegelbot v20200716.01)
Please talk to my owner @boegel if you notice you me acting stupid),
or submit a pull request to https://github.com/boegel/boegelbot fix the problem.
Test report by @ItIsI-Orient SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) node3701.slaking.os - Linux RHEL 8.4, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 3.6.8 See https://gist.github.com/0e3296e4f9ee74dc5ba6142219dd4442 for a full test report.
Test report by @bedroge FAILED Build succeeded for 0 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 22.04, x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.10.4 See https://gist.github.com/78b729178009076c6956e810e7133941 for a full test report.
Test report by @bedroge SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) pg-interactive.hpc.rug.nl - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 3.6.8 See https://gist.github.com/33f962162202fdcf123c663958711b4a for a full test report.
The installation of the Python extension fails on my Ubuntu system:
[25/26] Linking CXX static library lib/libraven.a
[26/26] Linking CXX shared module lib/ravenpy.cpython-310-x86_64-linux-gnu.so
error: [Errno 2] No such file or directory: '/data/eb/tmp/eb-aklw6_1o/pip-req-build-6c3ee0ie/build/lib.linux-x86_64-3.9/ravenpy_build/lib/ravenpy.cpython-39-x86_64-linux-gnu.so'
It looks like it's somehow mixing up the Python versions from the OS (3.10) and the module (3.9)...
-- Found PythonInterp: /usr/bin/python3.10 (found version "3.10.4")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.10.so
I've debugged this a bit more, and the issue seems to be caused by the way CMake is trying to find Python. This behavior has been changed in CMake 3.15: https://cmake.org/cmake/help/latest/policy/CMP0094.html#policy:CMP0094
With older versions/policies of CMake, it would just look for the newest Python it could find (and since my Ubuntu has Python 3.10, it prefers that one), with newer ones the location is preferred. We do use a newer CMake for this easyconfig, but because Raven's CMakeLists.txt has cmake_minimum_required(VERSION 3.11), it still falls back to the old policy. We can change this by either patching that line in CMakeLists.txt, or by explicitly enabling the new policy in setup.py; I'd prefer the latter. On top of that, we should also make sure that pybind uses CMake's FindPython functionality by setting PYBIND11_FINDPYTHON=ON.
Finally, I see that the installation pulls in some dependencies, see: https://github.com/lbcb-sci/raven/blob/1.8.1/Raven.deps.cmake It would be cleaner if we would list these as dependencies in the easyconfig. For most(/all?) of them, easyconfigs already seem to exist, they may only require a version bump.
Will take a look into that.
Also I am not sure how to enable the new policy in setup.py, could you point me in the right direction about solving this one?
Will take a look into that.
Also I am not sure how to enable the new policy in
setup.py, could you point me in the right direction about solving this one?
@ItIsI-Orient Via -DPython3_FIND_STRATEGY I guess, see https://cmake.org/cmake/help/latest/policy/CMP0094.html ?
We fixed a very similar problem with the Python being used for VTK in #16741 via -DPython3_EXECUTABLE, which we should also use here (and actually everywhere where CMake is being used and Python is included as a dependency, so in the CMakeMake easyblock, but that's a big hammer...)
Test report by @ItIsI-Orient SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) node3703.slaking.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 3.6.8 See https://gist.github.com/89f3570fd5c99e45fc384ccf907537b7 for a full test report.
@bedroge Could you please test, whether the issue you had is resolved?
Test report by @bedroge FAILED Build succeeded for 0 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 22.04, x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.10.6 See https://gist.github.com/206dd0cc6458400f3d2af95b759d14b0 for a full test report.
This doesn't seem to solve the issue, as it's only passed to the build of Raven itself, while the issue pops up while RavenPy is being built. In order to pass it to that (CMake) build as well, I guess we have to modify the setup.py (e.g. using something like what I used in a previous suggestion here: https://github.com/easybuilders/easybuild-easyconfigs/pull/15833#pullrequestreview-1045043403), which also calls cmake with some predefined arguments.
Test report by @bedroge SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 22.04, x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.10.6 See https://gist.github.com/73b29b6588e22bf6d03ba1f6df8dfe64 for a full test report.
So, this does seem to work (but I had already tried that a while ago). I'm not sure though if this is what @boegel meant, as it only sets -DPython_FIND_STRATEGY=LOCATION now, while @boegel was referring to -DPython3_EXECUTABLE . I also tried setting only the latter for ravenpy, but that didn't work, so it looks like we do need the first one as well (or maybe both to be sure?). This is a bit of a weird one anyway, as setup.py calls cmake itself, I'm not sure how usual this is.
@boegel any thoughts on what to do here?
I'm surprised that only setting -DPython3_EXECUTABLE isn't sufficient...
What does using -DPython_FIND_STRATEGY=LOCATION actually imply?
@ItIsI-Orient Can you take a detailed look at that, and try and see if you can figure out why it seems that -DPython_FIND_STRATEGY=LOCATION is required (too)?
I'm surprised that only setting
-DPython3_EXECUTABLEisn't sufficient... What does using-DPython_FIND_STRATEGY=LOCATIONactually imply?
From the CMake documentation
LOCATION: Stops lookup as soon as a version satisfying version constraints is founded.
The alternative strategy is for it to find the latest version.
-DPython3_EXECUTABLE is being set for Raven, but not for ravenpy.
And, as mentioned in https://github.com/easybuilders/easybuild-easyconfigs/pull/15833#issuecomment-1190059574, I think the issue is that the CMakeLists.txt file sets the minimum required CMake version to a rather old one, which also leads to old policies being used.
In this case it means that it will / seems to use VERSION instead of LOCATION for Python_FIND_STRATEGY; as Simon already mentioned, this means it will use the newest Python version it can find instead of looking for the first one (based on your $PATH I guess?) that satisfies the version constraints.
Test report by @bedroge SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 22.04, x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.10.6 See https://gist.github.com/7ec344fd25ef7bb885a1885d729ec361 for a full test report.
This version works for me. @boegel are you also okay with the current approach/version?
@bedroge: Request for testing this PR well received on login1
PR test command 'EB_PR=15833 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_15833 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!
- exit code: 0
- output:
Submitted batch job 9972
Test results coming soon (I hope)...
- notification for comment with ID 1380701776 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
Test report by @boegelbot FAILED Build succeeded for 0 out of 1 (1 easyconfigs in total) cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8 See https://gist.github.com/099788fc99a833910223e968a6e16848 for a full test report.
sanity check command pip check exited with code 1 (output: poetry 1.1.7 has requirement keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0", but you have keyring 23.9.3.
I don't understand where this version 23.9.3 comes from, the Python installation has 21.2.0...
@boegel: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster
PR test command 'EB_PR=15833 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_15833 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!
- exit code: 0
- output:
Submitted batch job 2251
Test results coming soon (I hope)...
- notification for comment with ID 1443740040 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
sanity check command pip check exited with code 1 (output: poetry 1.1.7 has requirement keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0", but you have keyring 23.9.3.I don't understand where this version 23.9.3 comes from, the Python installation has 21.2.0...
There was some stuff installed in the home directory of the bot that explains this:
[boegelbot@login1 ~]$ ml Python/3.9.6-GCCcore-11.2.0
[boegelbot@login1 ~]$ python3 -m pip list | grep keyring
keyring 23.9.3
[boegelbot@login1 ~]$ python3
Python 3.9.6 (default, Jul 14 2022, 08:55:00)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keyring
>>> keyring.__file__
'/home/boegelbot/.local/lib/python3.9/site-packages/keyring/__init__.py'
I cleaned that up, should be fine now...
Test report by @boegelbot SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8 See https://gist.github.com/b52f792e360fe538258b85028df64b5a for a full test report.
@boegel: Request for testing this PR well received on login1
PR test command 'EB_PR=15833 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_15833 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!
- exit code: 0
- output:
Submitted batch job 10358
Test results coming soon (I hope)...
- notification for comment with ID 1443865533 processed
Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).
Test report by @boegel SUCCESS Build succeeded for 1 out of 1 (1 easyconfigs in total) node3125.skitty.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8 See https://gist.github.com/7c1b0a19f7e28bf2de4510f47adb7997 for a full test report.