Why not using 3D data for training?

[oym1994]

Hi，Thanks for your great job!

In manipulation task, depth camera is a very indispensable sensor and is easy to apply. The question is why not use this kind of camera in your project and training procedure?

Thanks for your attention. I am always waiting for your kind response

[yangjiangeyjg]

This is a good question.

[ayzaan]

Hi, thanks for the question. Overall we tried to minimize the number of modalities, and ended up being able to learn good policies using only RGB. Perhaps this is because the action space is constrained to 2d.

[ayzaan]

It's possible to experiment with this in sim by producing a dataset that also outputs a depth image, using the provided oracle policies.

[oym1994]

It's possible to experiment with this in sim by producing a dataset that also outputs a depth image, using the provided oracle policies.

Hi, thanks for you kind response. What dose oracle(control) and script(lable) mean? (Please forgive my poor English terminology)

[Bailey-24]

because CLIP only can use RGB?