Get the whole 3d scene state (from a global POV)

[nikita-petrashen]

Hi! Is there any way to access the whole state of the scene through ai2thor Python interface? For example, in case of an agent which sees the scene from a global perspective, from which the scene is fully observable as a 3D structure.

If it's not possible now, could you please hint how this can be implemented?

Thanks!

[mattdeitke]

Hi @nikita-petrashen,

I'd recommend adding a 3rd party camera to the scene: https://ai2thor.allenai.org/ithor/documentation/environment-state/#add-camera.

To add a top-down camera to the scene, I'd recommend checking out this notebook: https://colab.research.google.com/drive/1GSIF78B62hNskyr-SYtUMgMdsZ7tJpJQ?usp=sharing

[nikita-petrashen]

Thanks for your reply! Sadly, this is not what I need. I'd like to be able to run the whole scene through a point cloud segmentation model. Is it possible to access mesh data of the scene through ai2thor or some sort of a Unity script? Thanks!

[nikita-petrashen]

I've ran trough the code of ai2thor but couldn't figure out where the 3d mesh data is converted to 2d renders (RGB, depth, semantic masks etc.).

[mattdeitke]

Thanks for your reply! Sadly, this is not what I need. I'd like to be able to run the whole scene through a point cloud segmentation model. Is it possible to access mesh data of the scene through ai2thor or some sort of a Unity script? Thanks!

Ahh, I see. We don't currently support returning the whole mesh of the scene from Unity. I'm not sure how easy this would be to support, but adding in @AlvaroHG, who might know the most on this.

I've ran trough the code of ai2thor but couldn't figure out where the 3d mesh data is converted to 2d renders (RGB, depth, semantic masks etc.).

This is done inside of Unity with cameras and shaders.

[nikita-petrashen]

Great, thanks! Could you please point out the place in the C# code where it happens? I believe it's somewhere inside the unity folder in the repo, but couldn't find where it happens.