[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137

[GeYuhong]

/usr/bin/env /home/PJLAB/geyuhong/anaconda3/envs/policydissect/bin/python /home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 39341 -- /home/PJLAB/geyuhong/policydissect/play/play_anymal.py *** Warning: failed to preload CUDA lib *** Warning: failed to preload PhysX libs Importing module 'gym_37' (/home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so) Setting GYM_USD_PLUG_INFO_PATH to /home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json PyTorch version 1.10.0+cu113 Device count 1 /home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/src/gymtorch Using /home/PJLAB/geyuhong/.cache/torch_extensions/py37_cu113 as PyTorch extensions root... Emitting ninja build file /home/PJLAB/geyuhong/.cache/torch_extensions/py37_cu113/gymtorch/build.ninja... Building extension module gymtorch... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module gymtorch... pygame 2.3.0 (SDL 2.24.2, Python 3.7.16) Hello from the pygame community. https://www.pygame.org/contribute.html Setting seed: 1 Not connected to PVD /buildAgent/work/99bede84aa0a52c2/source/physx/src/gpu/PxPhysXGpuModuleLoader.cpp (148) : internal error : libcuda.so!

[Warning] [carb.gym.plugin] Failed to create a PhysX CUDA Context Manager. Falling back to CPU. Physics Engine: PhysX Physics Device: cpu GPU Pipeline: disabled Backend TkAgg is interactive backend. Turning interactive mode on. /home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] [Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137 [Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors Traceback (most recent call last): File "/home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path pkg_name=pkg_name, script_name=fname) File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/home/PJLAB/geyuhong/policydissect/play/play_anymal.py", line 46, in play_anymal(args, activation_func=activation, map=forward_anymal, model_name="anymal_forward", parkour=False) File "/home/PJLAB/geyuhong/policydissect/policydissect/utils/isaacgym_utils.py", line 351, in play_anymal env, _ = task_registry.make_env(name=args.task, args=args, env_cfg=env_cfg) File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/utils/task_registry.py", line 103, in make_env headless=args.headless File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 51, in init super().init(cfg, sim_params, physics_engine, sim_device, headless) File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/base/legged_robot.py", line 87, in init self._init_buffers() File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 65, in _init_buffers super()._init_buffers() File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/base/legged_robot.py", line 572, in _init_buffers self.noise_scale_vec = self._get_noise_scale_vec(self.cfg) File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 126, in _get_noise_scale_vec noise_vec = torch.zeros_like(self.obs_buf[0]) RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

In /legged_gym/envs/base/legged_robot.py, line 488, i get '[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137', and after running the line 489, i get '[Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors'. These two errors prevent me from creating tensors on 'cuda'. I don't know how to solve this problem. Besides, I run the case in /isaacgym/python/examples/ and find that i will get the same error if the case has the code 'self.gym.refresh_dof_state_tensor(self.sim)' and 'self.gym.refresh_actor_root_state_tensor(self.sim)', e.g., franka_nut_bolt_ik_osc.py and franka_cube_ik_osc.py

[lizhiwen19900709]

I had the same problem