openai
SubprocVecEnv not working
I'm trying to use SubprocVecEnv to initialize multiple instances of one environment to use in ACKTR and A2C, however, there seem to be problems with SubprocVecEnv. Following this code, the following happens:
def make_env():
env = gym.make('Test-v0')
return env
env = gym.make('Test-v0') #Works, env initialization message is printed
env = make_env() #Works, env initialization message is printed
env = SubprocVecEnv([make_env]) #Doesn't work, code gets stuck
Meaning, env creation through the function works fine, the same goes for manual creation, however, when SubprocVecEnv is called the code gets stuck, nothing seems to happen and it doesn't stop running, nor does the environment get created.
I tried calling manually env = SubprocVecEnv([make_env]) and it yields a strange error:
env = SubprocVecEnv([make_env]) Traceback (most recent call last):
File "
", line 1, in env = SubprocVecEnv([make_env]) File "c:\users\x\baselines\baselines\common\vec_env\subproc_vec_env.py", line 58, in init observation_space, action_space = self.remotes[0].recv()
File "D:\Programs\Anaconda3\lib\multiprocessing\connection.py", line 250, in recv buf = self._recv_bytes()
File "D:\Programs\Anaconda3\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError
EOFError
Apparently even with the default envs ('Cartpole-V1' as depicted in the linked example) has the same problem. Both custom and default envs work fine for DummyVecEnv's though.
Edited: After some testing, the problem seems to be at:
observation_space, action_space = self.remotes[0].recv()
Is the process eternally waiting for a response?
Update: 10/10/2018:
So I tried digging up a bit deeper, and eventually came to the conclusion that a part of the problem is in calling functions in the default way:
def make_env_complete(seed, num_env, start_index=0):
def make_env(rank):
def _thunk():
env = gym.make('Test-v0')
env.seed(seed+rank)
env = Monitor(env, logger.get_dir() and os.path.join(logger.get_dir(), str(rank)))
return env
return _thunk
set_global_seeds(seed)
return SubprocVecEnv([make_env(i + start_index) for i in range(num_env)])
Does not work either, it just hangs in there forever. However, if return _thunk() is used instead, something happens, an error is thrown to be more specific:
File "C:/Users/X/Desktop/untitled0.py", line 54, in
make_env_complete(0, 1, 0) File "C:/Users/Horus/Desktop/untitled0.py", line 47, in make_env_complete return SubprocVecEnv([make_env(i + start_index) for i in range(num_env)])
File "c:\users\X\baselines\baselines\common\vec_env\subproc_vec_env.py", line 53, in init p.start()
File "D:\Programs\Anaconda3\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self)
File "D:\Programs\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj)
File "D:\Programs\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj)
File "D:\Programs\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child)
File "D:\Programs\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj)
File "c:\users\horus\baselines\baselines\common\vec_env_init_.py", line 176, in getstate return cloudpickle.dumps(self.x)
File "D:\Programs\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 895, in dumps cp.dump(obj)
File "D:\Programs\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 268, in dump return Pickler.dump(self, obj)
File "D:\Programs\Anaconda3\lib\pickle.py", line 409, in dump self.save(obj)
File "D:\Programs\Anaconda3\lib\pickle.py", line 521, in save self.save_reduce(obj=obj, *rv)
File "D:\Programs\Anaconda3\lib\pickle.py", line 634, in save_reduce save(state)
File "D:\Programs\Anaconda3\lib\pickle.py", line 476, in save f(self, obj) # Call unbound method with explicit self
File "D:\Programs\Anaconda3\lib\pickle.py", line 821, in save_dict self._batch_setitems(obj.items())
File "D:\Programs\Anaconda3\lib\pickle.py", line 847, in _batch_setitems save(v)
File "D:\Programs\Anaconda3\lib\pickle.py", line 521, in save self.save_reduce(obj=obj, *rv)
File "D:\Programs\Anaconda3\lib\pickle.py", line 634, in save_reduce save(state)
File "D:\Programs\Anaconda3\lib\pickle.py", line 476, in save f(self, obj) # Call unbound method with explicit self
File "D:\Programs\Anaconda3\lib\pickle.py", line 821, in save_dict self._batch_setitems(obj.items())
File "D:\Programs\Anaconda3\lib\pickle.py", line 847, in _batch_setitems save(v)
File "D:\Programs\Anaconda3\lib\pickle.py", line 476, in save f(self, obj) # Call unbound method with explicit self
File "D:\Programs\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 790, in save_file raise pickle.PicklingError("Cannot pickle files that are not opened for reading: %s" % obj.mode)
PicklingError: Cannot pickle files that are not opened for reading: wt
Hi @lhorus ! As you have already established, SubprocVecEnv's are nasty to debug, because if the worker process fails, the parent process is just left there waiting or gets an uninformative broken pipe. Making SubprocVecEnv more debug-friendly is on a TODO list; however, as a general rule of thumb for now, if the problem is with vecenv, I'd recommend replacing SubprocVecEnv with DummyVecEnv and debug that way. Now, in your particular case sounds like the error is actually specific to SubprocVecEnv - for some reason forking the process has troubles with open file descriptors. If you were to remove Monitor wrapper from make_env, would the resulting thing work?
I also faced the same issue and got it to work following the below hack, note that this is not a fix to the problem, it is a hack so that you can use SubProcEnv:
import os
from stable_baselines import logger
from stable_baselines.bench import Monitor
from stable_baselines.common import set_global_seeds
from stable_baselines.common.atari_wrappers import make_atari, wrap_deepmind
from stable_baselines.common.cmd_util import make_atari_env
from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines.common.policies import CnnPolicy
from stable_baselines.common.vec_env import VecFrameStack
from stable_baselines import ACER
def custom_atari_env(env_id, num_env, seed, wrapper_kwargs=None, start_index=0,
allow_early_resets=True):
"""
Create a wrapped, monitored SubprocVecEnv for Atari.
:param env_id: (str) the environment ID
:param num_env: (int) the number of environment you wish to have
in subprocesses
:param seed: (int) the inital seed for RNG
:param wrapper_kwargs: (dict) the parameters for wrap_deepmind function
:param start_index: (int) start rank index
:param allow_early_resets: (bool) allows early reset of the environment
:return: (Gym Environment) The atari environment
"""
if wrapper_kwargs is None:
wrapper_kwargs = {}
def make_env(rank):
def _thunk():
env = make_atari(env_id)
env.seed(seed + rank)
env = Monitor(env, logger.get_dir() and os.path.join(
logger.get_dir(), str(rank)),
allow_early_resets=allow_early_resets)
return wrap_deepmind(env, **wrapper_kwargs)
return _thunk
set_global_seeds(seed)
return DummyVecEnv([make_env(i + start_index) for i in range(num_env)])
def main():
env = custom_atari_env('BreakoutNoFrameskip-v4', num_env=4, seed=42)
env = make_atari_env('PongNoFrameskip-v4', num_env=4, seed=42)
env = VecFrameStack(env, n_stack=4)
model = ACER(CnnPolicy, env, verbose=1,
tensorboard_log='/private/home/akadian/teas-baselines/'
'simple-nav/stable-baselines-runs')
model.learn(total_timesteps=25)
model.save('acer.pong.model')
if __name__ == '__main__':
main()
@abhiskk from what I understand, your hack does not actually use SubprocVecEnv (it is replaced with DummyVecEnv). In OP's particular problem that should help indeed (because no forking / pickling is going on in DummyVecEnv); however, this fix will make things quite a bit slower.
It does use SubprocVecEnv, I just make a dummy call to custom_atari_env before making a call to the original make_atari_env.
env = custom_atari_env('BreakoutNoFrameskip-v4', num_env=4, seed=42)
env = make_atari_env('PongNoFrameskip-v4', num_env=4, seed=42)
The call to custom_atari_env I think sets something up in the background and the make_atari_env does not throw an error after that.
I was having a similar issue when creating a UnityEnv gym wrapper using SubprocVecEnv. For me, the fix was editing the default RPC server port because the port was already open. Editing: \mlagents\envs\environment.py -> UnityEnvironment -> init() to use a different default base_port.
A computer restart would have likely fixed this as well, I recommend trying a hard reset of your computer if you face a similar issue.
I have the same issue, this should get fixed. My code works perfectly fine with Dummy env but Subproc throws the exact same error
@Coronon could you provide the code you are running and error stacktrace? The baselines code has changed quite a bit since the issue was opened; it would be helpful to have a recent stacktrace.
I have the same issue, running on a Mac M3 `python import gymnasium as gym from gymnasium.wrappers.transform_observation import GrayscaleObservation
from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.vec_env import SubprocVecEnv
from stable_baselines3.common.monitor import Monitor
import ale_py
gym.register_envs(ale_py)
environment_name = "SpaceInvadersNoFrameskip-v4"
env = gym.make(environment_name) env = make_vec_env(environment_name, n_envs=8, vec_env_cls=SubprocVecEnv) # <---- error on this line `
Error given: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
To fix this issue, refer to the "Safe importing of main module"
section in https://docs.python.org/3/library/multiprocessing.html
Traceback (most recent call last):
File "/Users/.../.../.../PPO/experimentConfig.py", line 18, in
