GoalEnv for Robot-Supervisor scheme

[KelvinYang0320]

I integrate GoalEnv from gym/core.py with deepbots for Robot-Supervisor scheme. GoalEnv also inherite Env, but it imposes a required structure on the observation_space.

1. reset(self): self.observation_space must be a Goal-compatible observation space. It is required to contain at least three elements, namely observation, desired_goal, and achieved_goal. Refer to robot_env.py in gym.

2. compute_reward(self, achieved_goal, desired_goal, info): Since in our panda example, we need to get the observation to compute achieved_goal, and then we can compute the reward by using achieved_goal and desired_goal. Our original get_reward(action) is actually not that suitable for robotics control task. So I think we can let users implement their own step(), so I add raise NotImplementedError in step() function of RobotGoalSupervisor.

I think my coding style or implementation is not good enough. Any suggestions would be appreciated! :smile:

[tsampazk]

Thanks @KelvinYang0320 ! I'll review this PR in the following days.

[KelvinYang0320]

@tsampazk Thank you! Since it is mainly used for robotic control task in gym, I think I can make an deepworlds example with panda for GoalEnv demo.

[tsampazk]

@tsampazk Thank you! Since it is mainly used for robotic control task in gym, I think I can make an deepworlds example with panda for GoalEnv demo.

That sounds nice so we can actually have a usage example to make it more clear.

[KelvinYang0320]

As a general comment, i think we should also look into how the new GoalEnv classes interact with the emitter/receiver classes. Right now emitter/receiver classes seem to not have access to the GoalEnv stuff.

Yes, I am still trying to integrate GoalEnv with emitter/receiver. However, we might need to decide how to design the highest level class to support both gym.GoalEnv and gym.Env first, and then we can think about how to design emitter-receiver and robot-supervisor.

Thank you for reviewing the code!

[KelvinYang0320]

@tsampazk I have made an example in deepworlds to demo GoalEnv with Panda. Check this PR. Also, I have updated this GoalEnv PR for those we have discussed about. Thank you!