[Bug Report] Using `set_world_poses_from_view` with camera sensors

[elle-miller]

Hi there,

There is potentially a bug with setting the camera pose via set_world_poses_from_view, otherwise I am using it incorrectly.

This is my config & instantiation for the TiledCamera. I am using a personal DirectRLEnv.

# in the cfg
tiled_camera: TiledCameraCfg = TiledCameraCfg(
    prim_path="/World/envs/env_.*/Camera",
    offset=TiledCameraCfg.OffsetCfg(pos=(0.85, -1.10, 1.6), rot=(1,0,1,1), convention="world"),
    data_types=["rgb"],
    spawn=sim_utils.PinholeCameraCfg(
        focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.01, 2.5)
    ),
    width=80,
    height=80, debug_vis=True
)
eye = [0.9, 0.9, 0.6]
target = [0.3, 0, 0.2]

# ... and then in _setup_scene() 
self._tiled_camera = TiledCamera(self.cfg.tiled_camera)
eyes = torch.tensor(self.cfg.eye, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets = torch.tensor(self.cfg.target, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
self.scene.sensors["tiled_camera"] = self._tiled_camera

This setup leads to this error:

File "/home/emil/code/external/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 118, in __init__
    self._setup_scene()
  File "/home/emil/code/external/IsaacLab/IsaacLabExtension/exts/multimodal_gym/multimodal_gym/tasks/franka/lift.py", line 339, in _setup_scene
    self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
  File "/home/emil/code/external/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/camera/camera.py", line 339, in set_world_poses_from_view
    env_ids = self._ALL_INDICES
AttributeError: 'TiledCamera' object has no attribute '_ALL_INDICES'

My question is, am I using set_world_poses_from_view in the wrong place? If so, where should I call it from? I can get around this issue by putting the set_world_poses_from_view call in a later function call like _configure_gym_env_spaces(). However, this creates problems for me because the first timestep of collected image observations are from the TiledCameraCfg pose, instead of the updated eyes/targets one.

I want to use set_world_poses_from_view because it is more intuitive than setting the pos and quat in TiledCameraCfg.

System Info

• Isaac Sim Version: 4.1
• OS: Ubuntu 20.04
• Latest Isaac Lab release

Checklist

• [x] I have checked that there is no similar issue in the repo (required)
• [x] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• [ ] I can set the TiledCamera pose using eyes/targets instead of pos/quat, and this view renders correctly from timestep 0

[Mayankm96]

Seems like an implementation issue on our side. The variable _ALL_INDICES is not being defined. @pascal-roth can you take a look, please? The fix should be simple.

[pascal-roth]

Hi,

This is not an implementation error. The variable _ALL_INDICIES is available once the TiledCameras has been initialized. As the function set_world_poses_from_view directly updates the camera's view, it is impossible to use this function unless the camera has been initialized, i.e., when also _ALL_INDICIES is available.

As a fix, you can set initialize the scene and then use the set_world_poses_from_view before executing the first simulation step. This should result in your desired behavior.

Will close this issue now, as there is no implementation bug.

[elle-miller]

@pascal-roth Thank you for the reply!

Can you specify where set_world_poses_from_view should be placed in a direct RL env file, e.g. in the cartpole camera environment, without having to modify DirectRLEnv?

Like I said above, I can get around the error by putting it a later function call like _configure_gym_env_spaces(). However, this creates problems for me because the first timestep of collected image observations are from the TiledCameraCfg pose. I can also put it in a function like get_observations and use a flag to only call it once, but this is a messy solution. Would appreciate your thoughts here.

[elle-miller]

I am also experiencing very undefined behaviour when attempting to use this function with 2x tiled cameras.

eyes = torch.tensor(self.cfg.eye, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets = torch.tensor(self.cfg.target, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
eyes_front = torch.tensor(self.cfg.eyes_front, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets_front = torch.tensor(self.cfg.targets_front, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
self._tiled_camera_front.set_world_poses_from_view(eyes=eyes_front, targets=targets_front)

Strangely, setting eyes_front impacts the view of both _tiled_camera and _tiled_camera_front. Can this issue be re-opened? @pascal-roth

[elle-miller]

I can also report that when using two tiled cameras, setting the initial position in the config of one camera with pos and quat also seems to impact the position of the second camera. So perhaps the issue is not confined to set_world_poses_from_view.

[pascal-roth]

set_world_poses_from_view

For this part, I would place the function not inside the environment but instead after the environment has been initialized.

[pascal-roth]

I am also experiencing very undefined behaviour when attempting to use this function with 2x tiled cameras.

eyes = torch.tensor(self.cfg.eye, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets = torch.tensor(self.cfg.target, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
eyes_front = torch.tensor(self.cfg.eyes_front, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets_front = torch.tensor(self.cfg.targets_front, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
self._tiled_camera_front.set_world_poses_from_view(eyes=eyes_front, targets=targets_front)

Strangely, setting eyes_front impacts the view of both _tiled_camera and _tiled_camera_front. Can this issue be re-opened? @pascal-roth

@kellyguo11 integrated the tiled camera, she should be the person to help in this case.

[elle-miller]

set_world_poses_from_view

For this part, I would place the function not inside the environment but instead after the environment has been initialized.

Sorry, I'm just not understanding... Could you please state the filename/function name where I should be making this call? Do you mean outside of the environment file and in train.py?

# create isaac environment
env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)

# camera
eye = [1, 0, 0.3]
target = [0, 0, 0]
targets = torch.tensor(target, dtype=torch.float, device=env.device).repeat((env.num_envs, 1)) + env.scene.env_origins
eyes = torch.tensor(eye, dtype=torch.float, device=env.device).repeat((env.num_envs, 1)) + env.scene.env_origins
env._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)

This is not allowed (and also not ideal as want to configure different camera views for different envs).

AttributeError: accessing private attribute '_tiled_camera' is prohibited

[elle-miller]

@garylvov camera issue

[glvov-bdai]

@elle-miller I've created a small script that I think should address your issue.

You were most of the way there, you should just access the cameras like:

camera_list = [env.unwrapped.scene["tiled_camera1"], env.unwrapped.scene["tiled_camera2"]]
positions = torch.tensor([[2.5, 2.5, 2.5]], device="cuda:0").repeat(NUM_ENVS, 1)
targets = torch.tensor([[0.0, 0.0, 0.0]], device="cuda:0").repeat(NUM_ENVS, 1)

for camera in camera_list:
	camera.set_world_poses_from_view(positions, targets)
	print("set camera pose")

You can find the script here:https://gist.github.com/glvov-bdai/767ad9d76ea2a10b625fe7cbfbc21c20

Please let me know if that fixes your problem so I can close this issue

For more examples of existing camera things in our ecosystem, check out

https://github.com/isaac-sim/IsaacLab/blob/main/source/standalone/tutorials/04_sensors/benchmark_cameras.py

https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_camera_env_cfg.py

https://isaac-sim.github.io/IsaacLab/source/how-to/estimate_how_many_cameras_can_run.html

[glvov-bdai]

@elle-miller On further inspection, it looks like we're still running into https://github.com/isaac-sim/IsaacLab/issues/1070 for the time being. On my end, the images look slightly different due to noise levels, but it still seems like one is overriding another.

When that's resolved setting the view should work though!

[kellyguo11]

To answer the original question regarding set_world_poses_from_view, the API will need to be called after simulation starts, which happens after sim.reset() in DirectRLEnv. Your workaround with putting it in _configure_gym_env_spaces() will work, a cleaner approach is probably to put it in the task's __init__ method after calling super().__init__(). To update the camera pose immediately, you can also make a call to sim.render() after setting the pose.