can we get stereo camera feed?

[AGMahmoud]

I need to get stereo feed from the camera on the agent, is it possible?

[winthos]

This is not currently supported from the agent camera itself, but you may be able to get effectively the same results by manipulating a ThirdPartyCamera .

You can use the UpdateThirdPartyCamera action to manipulate this extra camera to be the same orientation as the agent's main camera, as well as offset it slightly from the agent's main camera to create a stereo effect. You can get the world space position of the agent's camera from the cameraPosition metadata returned after each event. The agent camera will also match the agent's rotation, so rotation from the Agent metadata can help get you the exact direction to match. And finally the horizon angle of the camera is also in the Agent metadata as cameraHorizon.

[AGMahmoud]

Thanks a lot for your reply, but I can not set the third camera horizon as it is not an argument of the "UpdateThirdPartyCamera" nor "AddThirdPartyCamera", can you help me with that? Also, the pose in the metadata is the pose of the agent not the camera is there an elevation offset I should take into account? Finally, adding a camera at the same pose, shows a part of the camera cover which cover a big part of the scene.

[winthos]

UpdateThirdPartyCamera does not have a distinct camera horizon parameter, as you can adjust the position and rotation vectors of the camera fully, unlike the agent camera who's y and z rotations are inherited from the agent body itself.

Note that the agent camera inherits its forward orientation based on the agent's rotation. This is because the agent camera is a child object of the agent, meaning all position and rotation changes done to the agent directly reflect on the agent camera because they are linked. You must use the y rotation of the agent in order to adjust the y rotation of the ThirdPartyCamera in order to make it match. Neither the agent nor the agent camera's z rotation should be altered.

is there an elevation offset I should take into account?

The cameraPosition metadata reports the global, world space position of the agent camera, which means it can be used to directly place the ThirdPartyCamera. This accounts for the offset of the camera within the agent's heirarchy, and even doing actions like Crouch and Stand which move the camera's position will have their changes reflected in cameraPosition.

This means if you wanted to place a ThirdPartyCamera exactly where the agent camera is, with the same horizon as the agent camera, you would do something the following:

controller.step(
    action="UpdateThirdPartyCamera",
    thirdPartyCameraId=0,
    position=dict(x=event.metadata['cameraPosition']['x'], y=event.metadata['cameraPosition']['y'], z=event.metadata['cameraPosition']['z']),
    rotation=dict(x=event.metadata['agent']['cameraHorizon'], y=event.metadata['agent']['rotation']['y'], z=0),
    fieldOfView=event.metadata['fov']
)

Note for a stereo camera type effect, you will need to add some offset to the event.metadata['cameraPosition'] vector returned in order to offset it based on the current forward orientation of the agent/agent camera. This will have to be done on your end. The above example places the ThirdPartyCamera exactly where the agent camera is.

Finally, adding a camera at the same pose, shows a part of the camera cover which cover a big part of the scene.

Regarding this, I do not fully understand what you are describing here, but if the agent's body geometry is blocking or obscuring part of the ThirdPartyCamera viewport, you can initialize the agent body to be invisible to all cameras (makeAgentsVisible initialization parameter), which should clear out any obscuring parts of the agent's geometry.

from ai2thor.controller import Controller

controller = Controller(
    agentMode="default",
    visibilityDistance=1.5,
    scene="FloorPlan212",

    # step sizes
    gridSize=0.25,
    snapToGrid=True,
    rotateStepDegrees=90,

    # image modalities
    renderDepthImage=False,
    renderInstanceSegmentation=False,

    # camera properties
    width=300,
    height=300,
    fieldOfView=90,
    
    # agent rendering flag
    makeAgentsVisible=False
)

[lvjihui]

Hello, how do I get Agent Camera Intrinsics？