Getting image observations from environment

[shivakanthsujit]

Hello,

If I am running on a headless server, is there anyway for me to get image observations from the environment so that I can generate/save a video of the agent's performance? Something like gym's env.render("rgb_array") that I can use when I have set headless=True.

Thanks!

[42jaylonw]

Hi,

What I did was create a camera sensor: camera_props = gymapi.CameraProperties() camera_props.width = 128 camera_props.height = 128 camera_handle = self.gym.create_camera_sensor(env_handle, camera_props) local_pos = gymapi.Vec3([1., -1, 1.]) target_pos = gymapi.Vec3([0., 0., 0.]) self.gym.set_camera_location(camera_handle, env_handle, local_pos, target_pos)

and load image whenever is needed: refresh state first: self.gym.render_all_camera_sensors(self.sim) self.gym.fetch_results(self.sim, True) self.gym.step_graphics(self.sim) get image tensor: img = self.gym.get_camera_image_gpu_tensor(self.sim, env_handle, cam_handle, gymapi.IMAGE_COLOR) img_tensor = gymtorch.wrap_tensor(img)

In this way, you can get image whenever headless is True or False

you can find more instructions in your isaacgym docs: $your_isaacgym/docs/programming/graphics.html

[song-hl]

envs = isaacgymenvs.make(
    seed=0,
    task="Ant",
    num_envs=2000,
    sim_device="cuda:0",
    rl_device="cuda:0",
    headless=True,
    virtual_screen_capture=False,
)
camera_props = gymapi.CameraProperties()
camera_props.width = 128
camera_props.height = 128


gym = gymapi.acquire_gym()
env_handle = envs.envs[0]

camera_handle = gym.create_camera_sensor(env_handle, camera_props)
local_pos = gymapi.Vec3(1.0, -1.0, 1.0)
target_pos = gymapi.Vec3(0., 0., 0.)
gym.set_camera_location(camera_handle, env_handle, local_pos, target_pos)

when I run camera_handle = self.gym.create_camera_sensor(env_handle, camera_props) camera_handle is -1 and gym.set_camera_location(camera_handle, env_handle, local_pos, target_pos) got the error

SetCameraLocation: Error: could not find camera with handle -1 in environment 0

Is any solution? Thanks a lot

[42jaylonw]

In my point of view, isaacgym does not support adding a camera sensor after the initialization or self.gym.prepare_sim(self.sim). In other words, if you are using the built-in making env making function isaacgymenvs.make you cannot add a camera sensor. You should either create your camera sensor earlier.

Or you can check example 2 from my wrapper, it's more easier to add a camera sensor and load image data. https://github.com/42jaylonw/shifu/

[Rowing0914]

hmm, this happens even before gym.prepare_sim. do you know why?

[mxllc]

It's things of the initialization of func gym.create_sim. You need to specify the parameter graphics_device. If this parameter is -1, then you would get following results.

SetCameraLocation: Error: could not find camera with handle -1 in environment 0

It's the parameter for GPU id for rendering. Here is a brief introduction of this function.