opensim-core
opensim-core copied to clipboard
opensim-org
ModuleNotFoundError No module named '_simbody'
Hello,
Following setup:
- win10
- anaconda
- visual studio 2019 (v16.8.6)
- cmake 3.19.5
- swigwin 3.0.12
- opensim 4.1
- opensim python wrapping
- simbody master commit 9daf0d26ea22551a5d10114bffd9b02fa51c53bc
Due to an error while installing opensim dependencies / simbody with latest Version of Visual Studio 2019 compiling simbody 3.7 release I installed simbody manually using simbody bleedingedge master (commit 9daf0d26ea22551a5d10114bffd9b02fa51c53bc). This includes a merge of following pull request, wich solves an issue with the Visual Studio 2019: https://github.com/simbody/simbody/pull/707. Installation (as described here: https://github.com/simbody/simbody/tree/9daf0d26ea22551a5d10114bffd9b02fa51c53bc#windows-using-visual-studio) was successfull (run_tests_parallel succeeded without any error). After that installed opensim 4.1 release from source. (including opensim python wrapping). Opensim installation with visual studio 2019 was successfull, (run all tests parallel without any error). Installing opensim, set path variable and install setup.py in the conda env were done.
After installing opensim opensim module is recognized in our python repository, but the simbody submodule is not:
Exception has occured: ModuleNotFoundError
No module named '_simbody'
Is that still an error of simbody or a problem with opensim? Any idea how to fix this?
kind regards, patrisity
@patrisity @aymanhab
I am having the same issue over and over.
If you upgrade numpy, it should fix the problem: pip install numpy --upgrade
However, it gets problematic in some cases. I am using tensorflow and opensim. Tensorflow needs numpy ~=1.19.2, and if you use the fix above, you end up with a newer numpy version that is not supported by tensorflow. If you use the numpy version that comes with tensorflow, you get a numpy version that does not work with opensim. I copy below the full error message, but here is I think an interesting line:
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Any idea about what to do? @carmichaelong @nickbianco
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
C:\OpenSim 4.2-2021-03-01-87bafa8\sdk\Python\opensim\simbody.py in swig_import_helper()
13 try:
---> 14 return importlib.import_module(mname)
15 except ImportError:
~\anaconda3\envs\py37\lib\importlib\__init__.py in import_module(name, package)
126 level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)
128
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in _gcd_import(name, package, level)
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in _find_and_load(name, import_)
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in _load_unlocked(spec)
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in module_from_spec(spec)
~\anaconda3\envs\py37\lib\importlib\_bootstrap_external.py in create_module(self, spec)
~\anaconda3\envs\py37\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
ImportError: numpy.core.multiarray failed to import
During handling of the above exception, another exception occurred:
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-d100d6ac66f2> in <module>
----> 1 import opensim
C:\OpenSim 4.2-2021-03-01-87bafa8\sdk\Python\opensim\__init__.py in <module>
----> 1 from .simbody import *
2 from .common import *
3 from .simulation import *
4 from .actuators import *
5 from .analyses import *
C:\OpenSim 4.2-2021-03-01-87bafa8\sdk\Python\opensim\simbody.py in <module>
15 except ImportError:
16 return importlib.import_module('_simbody')
---> 17 _simbody = swig_import_helper()
18 del swig_import_helper
19 elif _swig_python_version_info >= (2, 6, 0):
C:\OpenSim 4.2-2021-03-01-87bafa8\sdk\Python\opensim\simbody.py in swig_import_helper()
14 return importlib.import_module(mname)
15 except ImportError:
---> 16 return importlib.import_module('_simbody')
17 _simbody = swig_import_helper()
18 del swig_import_helper
~\anaconda3\envs\py37\lib\importlib\__init__.py in import_module(name, package)
125 break
126 level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)
128
129
ModuleNotFoundError: No module named '_simbody'
@antoinefalisse version 4.1 of opensim didn't include or require numpy, so this's expected
@antoinefalisse I don't think we need a late version of numpy for OpenSim bindings, if we use 1.19.2 that may work out better for everybody. Not sure how to enforce but would be good to find out first from @nickbianco if that's the case.
After glancing at the release notes for NumPy 1.20, downgrading to 1.19.2 should be fine. We should also update the README.md to include the required NumPy version and also consider implementing some basic error handling when importing the OpenSim python modules.
Somewhat related to this issue, while trying to setup Python on Mac I ran into this issue reported previously on the forum: https://simtk.org/plugins/phpBB/viewtopicPhpbb.php?f=91&t=12051&p=34082&start=0&view=.
To fix, I had add both <opensim_install_dir>/sdk/lib and <opensim_install_dir>/sdk/Simbody/lib on my DYLD_LIBRARY_PATH. This was from a March build of OpenSim 4.2, need to see if this issue persists in 4.3.
https://github.com/opensim-org/opensim-core/commit/8acecf6aff5a98e7884b031dc9dae5f3afb711c3
Hi @patrisity. We're still investigating this issue internally, but one potential fix is to add the shared libraries and Python modules somewhere in your path. If you're building from source, these are located at <opensim_install_dir>/sdk/Python/opensim. On Mac, I added this directory to the PYTHONPATH environment variable and this seemed to work. You could try doing the same on Windows (you might try added the directory to either the PATH or PYTHONPATH variables).
Hi,
thanks for your help.
build OpenSim in VS2019 and build the Simbody dependencies manually as described here didn't work for me. Unfortunately I can not tell why, I've tried several combinations of OpenSim (4.1, 4.2, master), swig (3.0.12 and 4.0.2) ... but didn't try your recent advice.
In the end I used VS 2017 for a new build of OpenSim 4.2 which is working fine. with Win10, Anaconda, Python3.7, cmake 3.19.8, swigwin 3.0.12, (and let opensim build simbody dependencies with master at https://github.com/simbody/simbody/tree/8cad2aa70b6c30410302c9d096ad3df1d2f73750 as defined in sim 4.2 superbuild by aymanhab here)
On Windows, using a clean Anaconda environment that isn't base (or root):
I got it working with creating and adding to the PYTHONPATH variable <opensim_install_dir>/sdk/Python (note that this is one folder up from the path that @nickbianco used above, and was the folder noted in another issue #2869). Adding the path to the "Path" variable doesn't seem to do anything.
I'm wondering if there's a way to add this automatically so that users don't have to add to their PYTHONPATH. I definitely don't know enough about how the setup.py file, but it seems like some stuff is added to sys.path (which PYTHONPATH seems to prepend to):
In Python on the command line, or in Spyder (both had the same behavior):
>>> import sys
>>> sys.path
['', 'C:\\repos\\opensim-core\\install\\sdk\\Python', 'C:\\Anaconda3\\envs\\opensim-core\\python37.zip', 'C:\\Anaconda3\\envs\\opensim-core\\DLLs', 'C:\\Anaconda3\\envs\\opensim-core\\lib', 'C:\\Anaconda3\\envs\\opensim-core', 'C:\\Anaconda3\\envs\\opensim-core\\lib\\site-packages', 'C:\\Anaconda3\\envs\\opensim-core\\lib\\site-packages\\opensim-4.3_2021_04_22_ffff7ec3c-py3.7.egg', 'C:\\Anaconda3\\envs\\opensim-core\\lib\\site-packages\\win32', 'C:\\Anaconda3\\envs\\opensim-core\\lib\\site-packages\\win32\\lib', 'C:\\Anaconda3\\envs\\opensim-core\\lib\\site-packages\\Pythonwin']
//cc @aymanhab
Some stackoverflow discussion that might be helpful regarding PYTHONPATH: https://stackoverflow.com/questions/12257747/permanently-adding-a-file-path-to-sys-path-in-python
TODOs to create temporary fix for current 4.2 users (from dev meeting conversation):
- Update the confluence page for installing the Python bindings on how to add/update the PYTHONPATH (https://simtk-confluence.stanford.edu/display/OpenSim/Scripting+in+Python)
- Create stickied post on the forum pointing to the update
FYI @carmichaelong @aymanhab
I updated the confluence page for Windows and Linux. @nickbianco the folder you mentioned above for Mac was different, and I'm not sure how the process of adding to PYTHONPATH works on Mac. Would you be able to update that section with a short blurb (probably similar to the Ubuntu blurb that was added)?
Unless you're building from source locally, the setup.py file is in the same location: <opensim-install-dir>/sdk/Python. Actually, the current instructions show where the directory is if building from source (except that /python2.7 should be updated to /python3.7).
Installing Anaconda with python 3.7 worked exactly as expected out of the box without touching PYTHONPATH. Now shifting to look into the differences between python 3.7 and 3.8
I updated the scripting page adding a note if using Python 3. I also added the instructions for setting the PYTHONPATH for Mac (copied from the Ubuntu section).
Official 4.2, python 3.7, no need for PYTHONPATH, just set PATH setup install works perfectly as expected
Per https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew PATH is not used to locate dll's on windows, instead an explicit call to e.g.
os.add_dll_directory("D:/OpenSim 4.3-2021-06-15-d117ea6/bin") does the trick. Now looking into automating the process
This change was introduced in version 3.8 and it mainly affects windows, will have to write specific documentation for this but would be good to know if they broke osx or linux
@adamkewley Since you looked into this in the past and investigated on multiple machines, your suggestions as to where to insert the line to add_dll_directory to help users would be appreciated.
Could we have configureOpenSim.py script similar to the Matlab one, and just have users point to the install directory? You can get the "bin" folder from there and run add_dll_directory.
EDIT: Seems like there's a way to do this, but it requires tkinter: https://stackoverflow.com/questions/9319317/quick-and-easy-file-dialog-in-python.
Actually, tkinter is only needed if we need the UI prompt. The Matlab script has this because it allows you to select the installation folder. But we could just use the "bin" folder associated with the installation that the user is running the configure script from. Also, configureOpenSim.py is maybe unnecessary if this could be included within setup.py.
@nickbianco Thanks for chiming in. A couple differences with Matlab:
- After Matlab setup, users don't need to do anything. Here they'll have to do it on each new session or environment unless we bake it into the init.py file or use some other hook (e.g. sitecustomizer.py, .pth files etc.)
- When users invoke 'python setup.py install' they invoke it from the distribution directory so we can get the path to the bin folder at that point and save it without asking the user for further error-prone browsing.
So, I believe there's a solution that is transparent to users just too many hooks, hacks and use cases that're making it hard to get my hands around the best maintainable process.
@aymanhab @carmichaelong @nickbianco I am having seemingly related but slightly different problem. I am trying to build OpenSim in a docker using anaconda & am having issues with the python bindings.
It seems like my build is going fine because when I run the opensim tests using ctest everything passes (including python tests).
I've posted my Dockerfile here: https://gist.github.com/gattia/ed9fa3fc444d5e185b797bedbe5e1d18
If I start a new docker and try python -c "import opensim" I get a similar issue to whats posted above:
ImportError: cannot import name '_simbody' from 'opensim' (/miniconda/lib/python3.7/site-packages/opensim-4.3.1-py3.7.egg/opensim/__init__.py)
Python can't find _simbody. I can fix this particular problem by importing the opensim version inside ~/opensim_install/sdk/Python. Either using the PYTHONPATH option above:
export PYTHONPATH=/root/opensim_install/sdk/Python:$PYTHONPATH
or :
cd ~/opensim_install/sdk/Python
python -c "import opensim"
or
python -c "import sys; sys.path.insert(0, '/root/opensim_install/sdk/Python'); import opensim"
However, once I do that, it brings another problem - now when I try to import opensim it can't find the libpython it needs:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/root/opensim_install/sdk/Python/opensim/__init__.py", line 6, in <module>
from .simbody import *
File "/root/opensim_install/sdk/Python/opensim/simbody.py", line 13, in <module>
from . import _simbody
ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
I can make it find the libpython by adding its location to my LD_LIBRARY_PATH:
export LD_LIBRARY_PATH=/miniconda/lib/:$LD_LIBRARY_PATH
Now, opensim imports fine. However, I dont think this is the "right" way to do it. This is fine in a docker container because I only plan to use one python environment anyways - but in normal day-to-day use we have effectively linked the libpython inside of /miniconda/lib/ to always be the one that is found - which would cause trouble if you ever tried to use another conda environment and sort of defeats the purpose of multiple python environments.
Any thoughts or recommendations?
Hi @gattia, there's a docker image with python installed here https://hub.docker.com/repository/docker/stanfordnmbl/opensim-python
It would be great if we can build on it using your work to make a docker container with conda as well.
If you're referring to creating different conda environments that use different opensim or python versions then I agree it's a problem however this is ultimately dictated by swig bindings which practically ties our bindings to a specific python version.
The long term plan to handle this has been to use different pypi packages for different versions but that hasn't materialized yet.
@aymanhab, thanks for the quick reply!
More than happy to help with that. The one I built is actually from a different fork of opensim-core (https://github.com/opensim-jam-org/opensim-core) but could be for the regular opensim-core by just changing a single line in the Dockerfile. Let me know what you think the easiest way for me to contribute this would be.
The issue I have with the hack I used to make opensim work with conda has to do with the fact that using the hack would affect all conda environments (regardless of if opensim is in them or not).
For some reason using the opensim version inside of: <opensim_install_location>/sdk/Python results in _simbody not being able to find the right libpython and to get _simbody to find the right one, I had to add it to the LD_LIBRARY_PATH using export LD_LIBRARY_PATH=/miniconda/lib/:$LD_LIBRARY_PATH ... the problem with that (in my view) is that now if I change my environment for another project, then /miniconda/lib is permanently in my LD_LIBRARY_PATH and the new environment will continue to find that version of libpython even if Im in an environment with a completely different version of python.
However, I've just found a post that had to do the same thing and someone suggested that LD_LIBRARY_PATH can be updated when a conda environment is activated (https://github.com/deepmind/acme/issues/47#issuecomment-966766327) and can even use the $CONDA_PREFIX environment variable to make sure it points to the right environment.
If anyone runs into the same problem I did - this is probably the right solution.
For anyone trying to set a conda environment up to add/remove LD_LIBRARY_PATH on activation/deactivation (probably me in the future), here's what I did I used: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux
as a guide.
- Activate the conda environment you've installed opensim
- Create files that are used/checked during activation/deactivation.
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
- Edit ./etc/conda/activate.d/env_vars.sh as follows:
export ORIG_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
- Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
export LD_LIBRARY_PATH=$ORIG_LD_LIBRARY_PATH
Then, when you conda activate env_name it will add $CONDA_PREFIX/lib to LD_LIBRARY_PATH and when you conda deactivate it will return the LD_LIBRARY_PATH to what it was before activation. The one downside is that this might cause issues if you change the LD_LIBRARY_PATH when an environment is activated (the change you made will be undone when you deactivate).
