symengine
PDM and Poetry can't install symengine properly
I am trying to run symengine in a PDM or Poetry project and while I can install it via "pdm add" or "poetry add" and I find it in the virtualenv site-packages, the lib directory is empty compared to my globally installed version with my OS python.
When I run python directly from the venv and import symengine I get:
Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/****/****/****/.venv/lib/python3.12/site-packages/symengine/__init__.py", line 12, in <module>
import symengine.lib.symengine_wrapper as wrapper
ModuleNotFoundError: No module named 'symengine.lib.symengine_wrapper'
Is this a problem with both Poetry and PDM or with symengine not picking something up?
I tested to create a venv with Python 3.12 and install symengine without using poetry and it works fine on my computer:
$ python3.12 --version
Python 3.12.0
[~/testing]
$ python3.12 -m venv py312venv
[~/testing]
$ cd py312venv
[~/testing/py312venv]
$ . ./bin/activate
(py312venv) [~/testing/py312venv]
$ pip install symengine
Collecting symengine
Obtaining dependency information for symengine from https://files.pythonhosted.org/packages/b5/58/d22492431cefd539a3655ec298752ad56a11c349bb32c668d2567172118f/symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata
Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (1.2 kB)
Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (39.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.3/39.3 MB 19.7 MB/s eta 0:00:00
Installing collected packages: symengine
Successfully installed symengine-0.11.0
[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
(py312venv) [~/testing/py312venv]
$ python
Python 3.12.0 (main, Nov 14 2023, 08:54:25) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
>>>
The files installed:
[~/testing/py312venv/lib/python3.12/site-packages]
$ tree symengine*
symengine
├── functions.py
├── __init__.py
├── lib
│ ├── __init__.py
│ ├── __pycache__
│ │ └── __init__.cpython-312.pyc
│ ├── pywrapper.h
│ ├── symengine.pxd
│ ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
│ └── symengine_wrapper.pxd
├── printing.py
├── __pycache__
│ ├── functions.cpython-312.pyc
│ ├── __init__.cpython-312.pyc
│ ├── printing.cpython-312.pyc
│ ├── sympy_compat.cpython-312.pyc
│ ├── test_utilities.cpython-312.pyc
│ └── utilities.cpython-312.pyc
├── sympy_compat.py
├── tests
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-312.pyc
│ │ ├── test_arit.cpython-312.pyc
│ │ ├── test_cse.cpython-312.pyc
│ │ ├── test_dict_basic.cpython-312.pyc
│ │ ├── test_eval.cpython-312.pyc
│ │ ├── test_expr.cpython-312.pyc
│ │ ├── test_functions.cpython-312.pyc
│ │ ├── test_lambdify.cpython-312.pyc
│ │ ├── test_logic.cpython-312.pyc
│ │ ├── test_matrices.cpython-312.pyc
│ │ ├── test_ntheory.cpython-312.pyc
│ │ ├── test_number.cpython-312.pyc
│ │ ├── test_pickling.cpython-312.pyc
│ │ ├── test_printing.cpython-312.pyc
│ │ ├── test_sage.cpython-312.pyc
│ │ ├── test_series_expansion.cpython-312.pyc
│ │ ├── test_sets.cpython-312.pyc
│ │ ├── test_solve.cpython-312.pyc
│ │ ├── test_subs.cpython-312.pyc
│ │ ├── test_symbol.cpython-312.pyc
│ │ ├── test_sympify.cpython-312.pyc
│ │ ├── test_sympy_compat.cpython-312.pyc
│ │ ├── test_sympy_conv.cpython-312.pyc
│ │ └── test_var.cpython-312.pyc
│ ├── test_arit.py
│ ├── test_cse.py
│ ├── test_dict_basic.py
│ ├── test_eval.py
│ ├── test_expr.py
│ ├── test_functions.py
│ ├── test_lambdify.py
│ ├── test_logic.py
│ ├── test_matrices.py
│ ├── test_ntheory.py
│ ├── test_number.py
│ ├── test_pickling.py
│ ├── test_printing.py
│ ├── test_sage.py
│ ├── test_series_expansion.py
│ ├── test_sets.py
│ ├── test_solve.py
│ ├── test_subs.py
│ ├── test_symbol.py
│ ├── test_sympify.py
│ ├── test_sympy_compat.py
│ ├── test_sympy_conv.py
│ └── test_var.py
├── test_utilities.py
└── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── INSTALLER
├── LICENSE
├── METADATA
├── RECORD
├── REQUESTED
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5
Yes, I got the same with just a normal venv.
What I notice is that several directories in the site-packages are missing in the pdm version:
symengine
├── functions.py
├── __init__.py
├── lib
│ ├── __init__.py
│ └── __pycache__
│ └── __init__.cpython-312.pyc
├── printing.py
├── __pycache__
│ └── __init__.cpython-312.pyc
├── sympy_compat.py
├── tests
│ ├── __init__.py
│ ├── test_arit.py
│ ├── test_cse.py
│ ├── test_dict_basic.py
│ ├── test_eval.py
│ ├── test_expr.py
│ ├── test_functions.py
│ ├── test_lambdify.py
│ ├── test_logic.py
│ ├── test_matrices.py
│ ├── test_ntheory.py
│ ├── test_number.py
│ ├── test_pickling.py
│ ├── test_printing.py
│ ├── test_sage.py
│ ├── test_series_expansion.py
│ ├── test_sets.py
│ ├── test_solve.py
│ ├── test_subs.py
│ ├── test_symbol.py
│ ├── test_sympify.py
│ ├── test_sympy_compat.py
│ ├── test_sympy_conv.py
│ └── test_var.py
├── test_utilities.py
└── utilities.py
The lib/ folder is empty, symengine.libs doesn't get created, and symengine-0.11.
Perhaps this is more a question for pdm and poetry than symengine? They seem to be missing the external libraries.
Could still be a problem with symengine. Does it work if you use poetry to install some other library that also uses compiled code, like for example numpy?
I just discovered that the lib/, symengine-0.11.0.dist-info, symengine.libs are placed in the lib64 directory.
.venv/lib
└── python3.12
└── site-packages
├── __pycache__
│ └── _virtualenv.cpython-312.pyc
├── symengine
│ ├── functions.py
│ ├── __init__.py
│ ├── lib
│ │ └── __init__.py
│ ├── printing.py
│ ├── sympy_compat.py
│ ├── tests
│ │ ├── __init__.py
│ │ ├── test_arit.py
│ │ ├── test_cse.py
│ │ ├── test_dict_basic.py
│ │ ├── test_eval.py
│ │ ├── test_expr.py
│ │ ├── test_functions.py
│ │ ├── test_lambdify.py
│ │ ├── test_logic.py
│ │ ├── test_matrices.py
│ │ ├── test_ntheory.py
│ │ ├── test_number.py
│ │ ├── test_pickling.py
│ │ ├── test_printing.py
│ │ ├── test_sage.py
│ │ ├── test_series_expansion.py
│ │ ├── test_sets.py
│ │ ├── test_solve.py
│ │ ├── test_subs.py
│ │ ├── test_symbol.py
│ │ ├── test_sympify.py
│ │ ├── test_sympy_compat.py
│ │ ├── test_sympy_conv.py
│ │ └── test_var.py
│ ├── test_utilities.py
│ └── utilities.py
├── _virtualenv.pth
└── _virtualenv.py
.venv/lib64
└── python3.12
└── site-packages
├── symengine
│ └── lib
│ ├── pywrapper.h
│ ├── symengine.pxd
│ ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
│ └── symengine_wrapper.pxd
├── symengine-0.11.0.dist-info
│ ├── AUTHORS
│ ├── LICENSE
│ ├── METADATA
│ ├── RECORD
│ ├── top_level.txt
│ └── WHEEL
└── symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5
Whereas numpy gets placed entirely in the lib64 site-packages not spread across.
And when I move all the files that are placed into the .venv/lib/python3.12/site-packages/ into the lib64 then everything works are expected.
Good find @martinruefenacht! I think this is the crucial problem. Now we just need to figure out if this is caused by the symengine wheel being composed in a faulty way or if this is the fault of the installation procedure.
Well I am not familiar with this stuff. I unpacked the numpy and symengine wheels to have a look. What stood out was that the symengine has a .data directory and that contains what I see in the lib/ instead of the lib64/.
symengine
└── lib
├── pywrapper.h
├── symengine.pxd
├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
└── symengine_wrapper.pxd
symengine-0.11.0.data
└── purelib
└── symengine
├── functions.py
├── __init__.py
├── lib
│ └── __init__.py
├── printing.py
├── sympy_compat.py
├── tests
│ ├── __init__.py
│ ├── test_arit.py
│ ├── test_cse.py
│ ├── test_dict_basic.py
│ ├── test_eval.py
│ ├── test_expr.py
│ ├── test_functions.py
│ ├── test_lambdify.py
│ ├── test_logic.py
│ ├── test_matrices.py
│ ├── test_ntheory.py
│ ├── test_number.py
│ ├── test_pickling.py
│ ├── test_printing.py
│ ├── test_sage.py
│ ├── test_series_expansion.py
│ ├── test_sets.py
│ ├── test_solve.py
│ ├── test_subs.py
│ ├── test_symbol.py
│ ├── test_sympify.py
│ ├── test_sympy_compat.py
│ ├── test_sympy_conv.py
│ └── test_var.py
├── test_utilities.py
└── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── LICENSE
├── METADATA
├── RECORD
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5
I just ran into this myself while symengine is being installed as part of a tox workflow.
I am running Fedora 39. It is odd that everything works fine outside of tox.
I performed a git bisection to find the first bad commit based on the fact that I knew v0.9.2 to be working. I made sure to always have the correct symengine version as well as compatible Cython installed. I used Python 3.10 since this is supported by all versions between 0.9 and 0.11.
The commit which introduced the failure to build as expected is: bc84086d60de038eb381c9e37c8b552a6c246ab5
I am not quite sure what to make of this but hopefully this helps in finding a solution.
No I am using pure pip for the installation. But the case in which it breaks is when this gets executed as part of tox
Strange. I use symengine myself with pip and tox and that works fine. I wonder if it is the same cause as in the poetry case.
I had a colleague point out to me, that it could possibly be related to how the OS (in this case Fedora) handles lib vs lib64 in Python. I did not find much that would be helpful on this, hence I did the bisection.
Okay, here a few more insights.
- that first bad commit was part of #402 which also includes an option to re-enable the previous behavior by setting
SYMENGINE_INSTALL_PY_FILES=ON. Insidesetup.pythis is hard-coded toOFFwhen callingcmake. Changing this toONdoes indeed make the latest commit (2223b99ffb91ff5f9f74aaf7a8071ab187caf955) work as expected with my tox - digging into the
libvslib64got me thinking, why things worked for me outside oftox. I noticed that myvenvhas a symlinklib64 -> libwhich is not whattoxis doing. The creation of this symlink seems to have changed at some point: see https://github.com/pypa/virtualenv/issues/1751 which is also linked to by apoetryissue, possibly explaining Martin's observations - and indeed, tox uses virtualenv > 20 which could explain this behavior
Even more information:
We seem to be running into the distinction of platlib and purelib which differs on Fedora: https://peps.python.org/pep-0427/#what-s-the-deal-with-purelib-vs-platlib
In setup.py, this project uses platlib consistently. With the symlink between lib and lib64 gone in virtualenv>=20 (which is what both, tox and poetry, use under the hood to manage their environment), this is likely the reason why symengine fails to install properly in both cases.
I have not worked with Python + cmake before, so the question remains on how to resolve this problem.
What I do not understand, is why the wheels that are built for symengine place their files in purelib.
I would expect them to use platlib given that root_is_pure = False is set inside of setup.py and that the C module is obviously not pure Python...
I just replicated the finding of @mrossinek with both tox and nox (3.8-3.12). Same problem as I found with poetry and pdm. This was all on Fedora. For fun I tested on Debian quickly with poetry and tox (3.11) and those worked.
Also have this error on rockylinux:8.8. Any fixes so far? Problem seems to be that pip install symengine places files in /usr/local/lib64 and not /usr/local/lib
My hotfix is to copy everything in my virtual environment to the /lib... manually.
I don't know how to fix this issue, I assume the setup.py has to be adjusted somehow to modify the packaging for fedora specifically?
This works
git clone https://github.com/symengine/symengine.py.git
cd symengine.py
git checkout v0.11.0
sed -i 's/-DSYMENGINE_INSTALL_PY_FILES=OFF/-DSYMENGINE_INSTALL_PY_FILES=ON/' setup.py
python setup.py install
Another data point: Problem persists for me on different fedora 39 and 40 platforms. It does not occur an an Arch linux platform.
Another data point to add to this discussion:
I was running into this issue and had discovered that poetry was not using the correct python version managed by my pyenv environment when building symengine. When installing the package, poetry was using my system python executable instead of the pyenv managed one (even though I had set the active python version to the one managed by pyenv). Specifying the $PATH to the correct version explicitly in the poetry environment resolved the issue.
