Request to add new baseline RVT or SAM2Act

[yqi19]

Hi, thanks for your amazing work.

May I ask do you have any plans to reproduce the RVT and SAM2Act (either of them will be fine) on current simulation? RVT: https://robotic-view-transformer-2.github.io/ SAM2Act: https://github.com/sam2act/SAM2Act

It seems that the current baseline lacks implementation of keypose-based policy, in RLBench we have EEMotionPlanning controller which automatically plans according to given keyposes. I do wonder how to implement such keypose-based policy on maniskill.

Thanks!

[StoneT2000]

We'd be very happy to accept such baselines! We don't have the manpower unfortunately at the moment to add this ourselves.

For keypose-based policies you can use the mplib library to plan poses and create an action wrapper around the environments. Essentially this gym.ActionWrapper will transform given pose actions into a sequence of low level joint control actions (pd_joint_pos mode) generated by a motion planning library like mplib. Then before completing the environment step this wrapper will step through all those low-level actions.

I think a keypose-based action wrapper is something I can help support. Baselines might be too much time on my end but I can provide suggestions.

[yqi19]

Thank you for your prompt reply. Yes, a keypose-based action wrapper can greatly help, I can try implement either RVT or SAM2Act based on it.