RL training not working

[charlo1998]

Hi @srivatsankrishnan , i'm having a similar issue to #1 : i'm able to connect, but the training won't start and I get a NotImplementedError. am I supposed to implement the step, reset etc. methods myself?

(workspace) C:\Users\charl\workspace\airlearning\airlearning-rl\run_time>python collect_data.py Using TensorFlow backend. C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) WARNING:tornado.general:Connect error on fd 4344: WSAECONNREFUSED Connected! Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

2022-05-13 10:06:15.588240: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2022-05-13 10:06:15.674282: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: name: NVIDIA GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 5.02GiB 2022-05-13 10:06:15.674451: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 2022-05-13 10:06:16.075145: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-05-13 10:06:16.075245: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 2022-05-13 10:06:16.076013: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 2022-05-13 10:06:16.076385: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3686 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5) Connected! Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

2022-05-13 10:06:16.299001: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 2022-05-13 10:06:16.299178: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-05-13 10:06:16.300228: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 2022-05-13 10:06:16.302869: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 2022-05-13 10:06:16.303480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3686 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5) Traceback (most recent call last): File "collect_data.py", line 104, in main() File "collect_data.py", line 100, in main runTask(task_el) File "collect_data.py", line 52, in runTask train_class.train(train_obj, env) File "C:\Users\charl\workspace\airlearning\airlearning-rl/algorithms/discrete/dqn\dqn_baselines.py", line 40, in train agent.learn(total_timesteps=settings.training_steps_cap) File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\deepq\dqn.py", line 170, in learn obs = self.env.reset() File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\common\base_class.py", line 523, in reset return self.venv.reset()[0] File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\common\vec_env\dummy_vec_env.py", line 45, in reset obs = self.envs[env_idx].reset() File "C:\Users\charl\workspace\lib\site-packages\gym\wrappers\order_enforcing.py", line 16, in reset return self.env.reset(**kwargs) File "C:\Users\charl\workspace\lib\site-packages\gym\core.py", line 74, in reset raise NotImplementedError NotImplementedError

[limshoonkit]

Hi @charlo1998 , @srivatsankrishnan I am encountering the same issue. Is there a fix for this?

Traceback (most recent call last): File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 104, in main() File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 100, in main runTask(task_el) File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 52, in runTask train_class.train(train_obj, env) File "C:\opt\AirLearning\airlearning\airlearning-rl/algorithms/discrete/dqn\dqn_baselines.py", line 41, in train agent.learn(total_timesteps=settings.training_steps_cap) File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\deepq\dqn.py", line 202, in learn new_obs, rew, done, _ = self.env.step(env_action) File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\vec_env\base_vec_env.py", line 94, in step return self.step_wait() File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\base_class.py", line 529, in step_wait actions, values, states, information = self.venv.step_wait() File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\vec_env\dummy_vec_env.py", line 36, in step_wait self.envs[env_idx].step(self.actions[env_idx]) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\wrappers\order_enforcing.py", line 37, in step return self.env.step(action) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\wrappers\env_checker.py", line 37, in step return env_step_passive_checker(self.env, action) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\utils\passive_env_checker.py", line 214, in env_step_passive_checker result = env.step(action) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\core.py", line 115, in step raise NotImplementedError NotImplementedError

[charlo1998]

Hi @limshoonkit, i was able to find a fix for this, although i don't know if this was the intended way to do it. If you search a bit the RL files, you'll see that the step method is declared like this: _step():

if i understood correctly, importing with "from gym_airsim.envs.airlearningclient import *" ignores the underscored methods, so i changed the declaration to step(): and it worked.

hope this helps

[limshoonkit]

Hi @charlo1998,

Thanks for the hint. I noticed you forked the project and made some changes. Did you have any luck on migrating the codebase from Tensorflow 1 to Tensorflow 2?

[charlo1998]

Hi @charlo1998,

Thanks for the hint. I noticed you forked the project and made some changes. Did you have any luck on migrating the codebase from Tensorflow 1 to Tensorflow 2?

unfortunately no, i stuck with tensorflow 1.8 and changed my CUDA and Cudnn drivers for compatibility

[qinglong0276]

Hello, how did you get to this point? I encountered an error "AirLearning could be compiled. Try Rebuilding from source manually" while trying to install the AirSim plugin for Air Learning in the first step of the Air Learning Environment Generator. Now, I don't know what to do for my next step. Can you teach me?