ray icon indicating copy to clipboard operation
ray copied to clipboard

[Bug] Windows access violation

Open avacaondata opened this issue 3 years ago • 3 comments

Search before asking

  • [X] I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

I was running a RL experiment with tune, and after the first training iteration I received the following error.

(pid=15328) 2022-02-04 07:42:47,493     INFO ppo.py:167 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting simple_optimizer=True if this doesn't work for you.
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff2d216c8f0f96d6df7ca28b7d01000000 Worker ID: f8fa879b5daada651392f3196ab44593ea31ad577b98c0cd47e79e7c Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59650 Worker PID: 17760
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff5831bd6fc9ebeff194ac47ac01000000 Worker ID: 4eccbb46a8965b98fe22dd66ff72df9f39279959e17be0e07bf530d8 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59682 Worker PID: 9436
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff7c9f0ecccd35b252e44e1ec501000000 Worker ID: 56ab92d164ca4ef8c0de0e2ae19030a12cb6a4a67c52c81d59b193ae Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59613 Worker PID: 17588
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffded776e8f0ed90e430f466e901000000 Worker ID: ea1f6f1217d7b9845f2749872f46d3d1161fe44b99e7d35cad5a6ee6 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59709 Worker PID: 20072
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff7aab45ba237795654c5e915401000000 Worker ID: 64b537e96a0e2d0d30ca4ecf6c62f09030cbc5e1a751902c0cb8d002 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59661 Worker PID: 17184
2022-02-04 07:42:59,570 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff34beadb82d4c81163a4e4bde01000000 Worker ID: 0f1bb74ff1eb7c65f8713a93d631d88023b83f9c3b063a426dc46114 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59675 Worker PID: 17572
2022-02-04 07:42:59,586 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff5db86521ce4a2e3da203cf7b01000000 Worker ID: f05aaccc707a8adc432a370b3f888f5021cfe9e300efbe5d2c1f8d97 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59695 Worker PID: 3512
2022-02-04 07:42:59,586 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff854ed36be7eaf7a8d25731d301000000 Worker ID: ca07ad03237dd67dfc691caa118fcd06af1905e2e8082b53d2da3292 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59641 Worker PID: 18324
2022-02-04 07:42:59,586 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff77e392568fe35638bda6084c01000000 Worker ID: 719ff219775b11d333991c5eba5870bb4628d9eb33d099f6b9d1858c Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59694 Worker PID: 2372
2022-02-04 07:42:59,602 WARNING worker.py:1228 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff62291d52ff4c2d3d7cc9095901000000 Worker ID: 5c4f8292a3ba6fb132ca33c411b9187fa8487a4e2e9860a8ea4a27c0 Node ID: 14681f2b73c425545882ccaa1d943d3d2ffb1e56ad05a6c82eef8511 Worker IP address: 127.0.0.1 Worker port: 59660 Worker PID: 19664
(pid=15328) 2022-02-04 07:42:59,570     ERROR worker.py:425 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=15328, ip=127.0.0.1)
(pid=15328)   File "python\ray\_raylet.pyx", line 565, in ray._raylet.execute_task
(pid=15328)   File "python\ray\_raylet.pyx", line 569, in ray._raylet.execute_task
(pid=15328)   File "python\ray\_raylet.pyx", line 519, in ray._raylet.execute_task.function_executor
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576, in actor_method_executor
(pid=15328)     return method(__ray_actor, *args, **kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451, in _resume_span
(pid=15328)     return method(self, *_args, **_kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer_template.py", line 137, in __init__
(pid=15328)     Trainer.__init__(self, config, env, logger_creator)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer.py", line 623, in __init__
(pid=15328)     super().__init__(config, logger_creator)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\tune\trainable.py", line 107, in __init__
(pid=15328)     self.setup(copy.deepcopy(self.config))
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451, in _resume_span
(pid=15328)     return method(self, *_args, **_kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer_template.py", line 147, in setup
(pid=15328)     super().setup(config)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer.py", line 776, in setup
(pid=15328)     self._init(self.config, self.env_creator)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451, in _resume_span
(pid=15328)     return method(self, *_args, **_kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer_template.py", line 176, in _init
(pid=15328)     num_workers=self.config["num_workers"])
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451, in _resume_span
(pid=15328)     return method(self, *_args, **_kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\agents\trainer.py", line 864, in _make_workers
(pid=15328)     logdir=self.logdir)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 89, in __init__
(pid=15328)     lambda p, pid: (pid, p.observation_space, p.action_space)))
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\client_mode_hook.py", line 105, in wrapper
(pid=15328)     return func(*args, **kwargs)
(pid=15328)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 1627, in get
(pid=15328)     raise value
(pid=15328) ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
(pid=9436) Windows fatal exception: access violation
(pid=9436)
(pid=9436) Stack (most recent call first):
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=9436)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=9436)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=17588) Windows fatal exception: access violation
(pid=17588)
(pid=17588) Stack (most recent call first):
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=17588)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=17588)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=19664) Windows fatal exception: access violation
(pid=19664)
(pid=19664) Stack (most recent call first):
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=19664)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=19664)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=17184) Windows fatal exception: access violation
(pid=17184)
(pid=17184) Stack (most recent call first):
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=17184)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=17184)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=18324) Windows fatal exception: access violation
(pid=18324)
(pid=18324) Stack (most recent call first):
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=18324)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=18324)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=17572) Windows fatal exception: access violation
(pid=17572)
(pid=17572) Stack (most recent call first):
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=17572)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=17572)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=2372) Windows fatal exception: access violation
(pid=2372)
(pid=2372) Stack (most recent call first):
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=2372)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=2372)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=17760) Windows fatal exception: access violation
(pid=17760)
(pid=17760) Stack (most recent call first):
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=17760)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=17760)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=20072) Windows fatal exception: access violation
(pid=20072)
(pid=20072) Stack (most recent call first):
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=20072)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=20072)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=3512) Windows fatal exception: access violation
(pid=3512)
(pid=3512) Stack (most recent call first):
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69 in __init__
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 1040 in _make_engine
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 811 in __init__
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 482 in _read
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\io\parsers\readers.py", line 586 in read_csv
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\pandas\util\_decorators.py", line 311 in wrapper
(pid=3512)   File "C:\Users\Usuario\Documents\CryptoRL\mysimple_env.py", line 95 in reset
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1524 in _validate_env
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 460 in __init__
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\util\tracing\tracing_helper.py", line 451 in _resume_span
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\_private\function_manager.py", line 576 in actor_method_executor
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\worker.py", line 425 in main_loop
(pid=3512)   File "C:\Users\Usuario\anaconda3\envs\cryptorl_ray18\lib\site-packages\ray\workers/default_worker.py", line 218 in <module>
(pid=9396) Windows fatal exception: access violation
(pid=9396)
(pid=12016) Windows fatal exception: access violation
(pid=12016)
(pid=11200) Windows fatal exception: access violation
(pid=11200)
(pid=10164) Windows fatal exception: access violation
(pid=10164)
forrtl: error (200): program aborting due to control-C event

Stack trace terminated abnormally.
Windows fatal exception: access violation

Versions / Dependencies

Python 3.7 Windows 10 Ray 1.8

Reproduction script

https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_ppo_example.py

Anything else

No response

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

avacaondata avatar Feb 04 '22 10:02 avacaondata

Thanks for the report, can you check if this is gone in Ray 1.9.2? This should have been fixed.

pcmoritz avatar Feb 04 '22 10:02 pcmoritz

If it still happens in 1.9.2, can you post a reproduction, i.e. which script are you running?

pcmoritz avatar Feb 04 '22 10:02 pcmoritz

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

  • If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
  • If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

stale[bot] avatar Aug 10 '22 03:08 stale[bot]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

stale[bot] avatar Sep 20 '22 18:09 stale[bot]