softlearning
softlearning copied to clipboard
rail-berkeley
Import error. Trying to rebuild mujoco_py.
$ softlearning run_example_local examples.development \
> --universe=gym \
> --domain=HalfCheetah \
> --task=v3 \
> --exp-name=my-sac-experiment-1 \
> --checkpoint-frequency=1000
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING: Logging before flag parsing goes to stderr.
I0418 01:08:50.825603 140032189581056 acceleratesupport.py:13] OpenGL_accelerate module loaded
I0418 01:08:50.832047 140032189581056 arraydatatype.py:270] Using accelerated ArrayDatatype
I0418 01:08:51.017610 140032189581056 __init__.py:34] MuJoCo library version is: 200
2019-04-18 01:08:51,105 INFO node.py:439 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-18_01-08-51_27162/logs.
2019-04-18 01:08:51,211 INFO services.py:364 -- Waiting for redis server at 127.0.0.1:20587 to respond...
2019-04-18 01:08:51,320 INFO services.py:364 -- Waiting for redis server at 127.0.0.1:52635 to respond...
2019-04-18 01:08:51,321 INFO services.py:761 -- Starting Redis shard with 10.0 GB max memory.
2019-04-18 01:08:51,337 WARNING services.py:1301 -- Warning: Capping object memory store to 20.0GB. To increase this further, specify `object_store_memory` when calling ray.init() or ray start.
2019-04-18 01:08:51,337 INFO services.py:1449 -- Starting the Plasma object store with 20.0 GB memory using /dev/shm.
2019-04-18 01:08:51,885 INFO tune.py:139 -- Did not find checkpoint file in /home/yrli/ray_results/gym/HalfCheetah/v3/2019-04-18T01-08-51-my-sac-experiment-1.
2019-04-18 01:08:51,885 INFO tune.py:145 -- Starting a new experiment.
2019-04-18 01:08:51,892 INFO web_server.py:241 -- Starting Tune Server...
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/56 CPUs, 0/8 GPUs
Memory usage on this node: 4.8/270.1 GB
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 56/56 CPUs, 0/8 GPUs
Memory usage on this node: 4.9/270.1 GB
Result logdir: /home/yrli/ray_results/gym/HalfCheetah/v3/2019-04-18T01-08-51-my-sac-experiment-1
Number of trials: 1 ({'RUNNING': 1})
RUNNING trials:
- id=f24a78d2-seed=4956: RUNNING
(pid=27322)
(pid=27322) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=27322) For more information, please see:
(pid=27322) * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=27322) * https://github.com/tensorflow/addons
(pid=27322) If you depend on functionality not listed there, please file an issue.
(pid=27322)
(pid=27322) 2019-04-18 01:08:55.340353: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=27322) Using seed 4956
(pid=27322) 2019-04-18 01:08:55.424360: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=27322) 2019-04-18 01:08:55.424399: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: 64.site
(pid=27322) 2019-04-18 01:08:55.424408: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: 64.site
(pid=27322) 2019-04-18 01:08:55.424460: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.104.0
(pid=27322) 2019-04-18 01:08:55.424498: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.104.0
(pid=27322) 2019-04-18 01:08:55.424507: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.104.0
(pid=27322) 2019-04-18 01:08:55.426316: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400060000 Hz
(pid=27322) 2019-04-18 01:08:55.429028: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5d18810 executing computations on platform Host. Devices:
(pid=27322) 2019-04-18 01:08:55.429054: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
(pid=27322) Import error. Trying to rebuild mujoco_py.
(pid=27322) Compiling /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/cymj.pyx because it depends on /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/pxd/mujoco.pxd.
(pid=27322) [1/1] Cythonizing /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/cymj.pyx
(pid=27322) running build_ext
(pid=27322) building 'mujoco_py.cymj' extension
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/gl
(pid=27322) gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py -I/home/yrli/.mujoco/mujoco200/include -I/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/numpy/core/include -I/home/yrli/anaconda3/envs/softlearning/include/python3.6m -c /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/cymj.c -o /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/cymj.o -fopenmp -w
(pid=27322) gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py -I/home/yrli/.mujoco/mujoco200/include -I/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/numpy/core/include -I/home/yrli/anaconda3/envs/softlearning/include/python3.6m -c /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/gl/osmesashim.c -o /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/gl/osmesashim.o -fopenmp -w
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/lib.linux-x86_64-3.6
(pid=27322) creating /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/lib.linux-x86_64-3.6/mujoco_py
(pid=27322) gcc -pthread -shared -L/home/yrli/anaconda3/envs/softlearning/lib -Wl,-rpath=/home/yrli/anaconda3/envs/softlearning/lib,--no-as-needed -L/home/yrli/anaconda3/envs/softlearning/lib -Wl,-rpath=/home/yrli/anaconda3/envs/softlearning/lib,--no-as-needed /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/cymj.o /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/temp.linux-x86_64-3.6/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/gl/osmesashim.o -L/home/yrli/.mujoco/mujoco200/bin -L/home/yrli/anaconda3/envs/softlearning/lib -Wl,--enable-new-dtags,-R/home/yrli/.mujoco/mujoco200/bin -lmujoco200 -lglewosmesa -lOSMesa -lGL -lpython3.6m -o /home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/generated/_pyxbld_2.0.2.0_36_linuxcpuextensionbuilder/lib.linux-x86_64-3.6/mujoco_py/cymj.cpython-36m-x86_64-linux-gnu.so -fopenmp
2019-04-18 01:09:53,990 ERROR trial_runner.py:426 -- Error processing event.
Traceback (most recent call last):
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 389, in _process_events
result = self.trial_executor.fetch_result(trial)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 252, in fetch_result
result = ray.get(trial_future[0])
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2288, in get
raise value
ray.exceptions.RayTaskError: ray_ExperimentRunner:train() (pid=27322, host=64.site)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/mujoco/mujoco_env.py", line 11, in <module>
import mujoco_py
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/__init__.py", line 3, in <module>
from mujoco_py.builder import cymj, ignore_mujoco_warnings, functions, MujocoException
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/builder.py", line 503, in <module>
cymj = load_cython_ext(mujoco_path)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/builder.py", line 106, in load_cython_ext
mod = load_dynamic_ext('cymj', cext_so_path)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/mujoco_py/builder.py", line 124, in load_dynamic_ext
return loader.load_module()
ImportError: dlopen: cannot load any more object with static TLS
During handling of the above exception, another exception occurred:
ray_ExperimentRunner:train() (pid=27322, host=64.site)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 150, in train
result = self._train()
File "/data1/yrli/softlearning/examples/development/main.py", line 77, in _train
self._build()
File "/data1/yrli/softlearning/examples/development/main.py", line 44, in _build
get_environment_from_params(environment_params['training']))
File "/data1/yrli/softlearning/softlearning/environments/utils.py", line 33, in get_environment_from_params
return get_environment(universe, domain, task, environment_kwargs)
File "/data1/yrli/softlearning/softlearning/environments/utils.py", line 24, in get_environment
return ADAPTERS[universe](domain, task, **environment_params)
File "/data1/yrli/softlearning/softlearning/environments/adapters/gym_adapter.py", line 66, in __init__
env = gym.envs.make(env_id, **kwargs)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183, in make
return registry.make(id, **kwargs)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125, in make
env = spec.make(**kwargs)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 88, in make
cls = load(self._entry_point)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 17, in load
mod = importlib.import_module(mod_name)
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/mujoco/__init__.py", line 1, in <module>
from gym.envs.mujoco.mujoco_env import MujocoEnv
File "/home/yrli/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/mujoco/mujoco_env.py", line 13, in <module>
raise error.DependencyNotInstalled("{}. (HINT: you need to install mujoco_py, and also perform the setup instructions here: https://github.com/openai/mujoco-py/.)".format(e))
gym.error.DependencyNotInstalled: dlopen: cannot load any more object with static TLS. (HINT: you need to install mujoco_py, and also perform the setup instructions here: https://github.com/openai/mujoco-py/.)
Unfortunately I have not seen this error before. How did you install the environment? Are you on the latest master?
Unfortunately I have not seen this error before. How did you install the environment? Are you on the latest master?
yes, I cloned the latest master. and I just change tensorflow to tensorflow-gpu. For tensorflow, I have the same log except for the cuda things. Can I switch you fork of mujoco to the official one and try?
Yeah, could you try setting mujoco-py==v1.50.1.0 and gym=0.2.0 in the requirements.txt and try again? Note that you need to have mujoco 1.5 installed.
Let me know how that goes!
change gym==0.2.0 in setup.py and requirement.txt get error as follow: from mujoco_py.mjlib import mjlib ModuleNotFoundError: No module named 'mujoco_py.mjlib'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/XXXXX/anaconda3/bin/softlearning", line 11, in
while use gym=0.18.0, error same as this issue:
pid=26386) 2020-12-28 16:00:21.925675: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
How can I resolve the problem
The latest error seems like a version mismatch between mujoco-py and gym versions. Would you mind checking which version of mujoco-py those specific version of gym requires and then try explicitly installing those? Or maybe running pip install -U gym==0.18.0 mujoco-py (with -U) might automatically upgrade mujoco-py at the same time?