Question - how to handle additional sensory input

[tlpss]

Hi guys, sorry to bother you again :wink:
and thanks for your work, I'm very excited by Lerobot!

I'm currently collecting some teleop data where the robot has tactile sensors on the fingertips, as well as a FT sensor on the wrist and I was wondering how I would integrate this best into a Lerobot Dataset.

One way would be to concatenate them into the observation.state, as this is the hardcoded location for non-image observations. But I want to train both with and without the tactile sensors and FT sensors as inputs to quantify the benefits of the other sensors, so I would then have to make separate datasets for each sensor combination which feels cumbersome.

Are there any plans in the near future to support 'dynamic configuration' of the state inputs for the policies? Or is my best option to just create different datasets for each combination?

[alexander-soare]

@tlpss thanks for bringing this up. It makes a lot of sense and we don't have it yet, but conceptually I think it could be very straight forward.

@Cadene: what if we do something similar as we've done with images. If it's one image it's "observation.image". If there are multiple we have "observation.images.x", "observation.images.y" etc. So with state vectors we can have: "observation.states.x", "observation.states.y" etc

[tlpss]

@aliberts @Cadene I'm still interested in this feature. If no one else is working on it, I might create a PR.

Any suggestions or questionsbefore I start? I would implement it as suggested by @alexander-soare

[tlpss]

@aliberts , one more ping? If not I will try to implement when I find some time

[aliberts]

Hi @tlpss, sorry for the late reply. I'm indeed refactoring some of our policies code and in doing so, I'm decoupling datasets keys from the policies (to not have these hardcoded). There will still be some form of hardcoded mapping in the near future but that mapping will be handled outside the policy, at the factory stage.

The core idea from Alexander of stacking inputs of similar type should still work though so if you want to draft a PR, we can then see how to implement your changes in this refactor.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity (6 months). It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity. Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.