panda_mujoco
panda_mujoco copied to clipboard
JeanElsner
Add joint and wrist torque sensors.
Hi @kevinzakka,
Your repository dexterity is really awesome, I've started exploring it recently, it is fun to use and I'm learning quite a bit from your code. Thank you for making this publicly available.
One question I had on the changes in this pr is when do you think one should add sensors via PyMJCF vs the MJCF xml file directly? I've seen torque sensors being added via PyMJCF more often and hence I wished to ask the question (it is worth noting that I am only getting up to speed on using dm-control more regularly, I am starting to use the panda MJCF file from this repo but have looked to add sensors separately via PyMJCF which is why I am asking).
Also, a secondary thought, when preparing repositories for MuJoCo entities, should they also include a PyMJCF file? I'd be happy to work towards adding one to this repo if you feel that it is worthwhile?
@peterdavidfagan Hey Peter, thanks for the kind words, I'm glad you're enjoying dexterity!
When do you think one should add sensors via PyMJCF vs the MJCF xml file directly?
That's a great question! I think it really depends on the codebase. On the one hand, it's nice to have a single XML file with all the bells and whistles. This is great for folks who don't necessarily use dm_control's PyMJCF, which I felt was the case for this repository. On the other hand, I really like the flexibility you get with PyMJCF. You can add or remove things to the XML file dynamically which is great for customization. For example, imagine you wanted to experiment with different actuation modes for the Panda arm. PyMJCF lets you do that pretty easily, but you'd need different XML files or commented out blocks otherwise which isn't great. dm_robotics does this with their sawyer arm if you're curious to see an example.
Also, a secondary thought, when preparing repositories for MuJoCo entities, should they also include a PyMJCF file?
That being said, I haven't figured out yet what's the best tradeoff for sharing high-quality models with the research community. My suspicion is that single source of truth XML files will be the way to go, and then you can have PyMJCF on top to tune the model to your own hardware or your own needs.
I'd be happy to work towards adding one to this repo if you feel that it is worthwhile?
I think it would be nice to have one for this repo, but I don't have a say in it since it's @JeanElsner 's repo :)
On the one hand, it's nice to have a single XML file with all the bells and whistles. This is great for folks who don't necessarily use dm_control's PyMJCF, which I felt was the case for this repository. On the other hand, I really like the flexibility you get with PyMJCF.
This makes sense, and I guess this is the case for this repository in its current state. I agree about the flexibility of PyMJCF, I am only starting to use it but it is quite useful for customization.
dm_robotics does this with their sawyer arm if you're curious to see an example.
Oh nice, I hadn't actually looked into this example yet, this is very useful thank you for linking this code.
My suspicion is that a single source of truth XML files will be the way to go, and then you can have PyMJCF on top to tune the model to your own hardware or your own needs.
The above makes sense, so the source of truth XML should contain the standard version of the robot and PyMJCF is reserved for customization. I had started to along the lines of keeping only the basics in the XML and adding additional details through PyMJCF but in hindsight for my use case having these predefined in the original XML works well.
I think it would be nice to have one for this repo, but I don't have a say in it since it's @JeanElsner 's repo :)
Oh yeah lol that is a fair point. @JeanElsner is adding customizable actuators with PyMJCF something you would see as valuable for this repo? I am happy to spend time adding this feature.
Hi guys,
sorry for the late reply, I was on vacation. But I'm quite invested in this :-) I totally get your point and I've also been considering this a lot recently. I'm not sure what the best way to share MuJoCo models with the research community is yet. This pull request is actually a good example of this. Allow me elaborate a bit.
I think that providing a single XML file with assets is the most general approach. However, there are some aspects that currently cannot be encoded in the model alone but are very useful to have. In fact, this is why I provided an example demonstrating a simple task-space controller. As for the sensors, the real robot only has link-side joint torque sensors. External joint torques and especially wrenches are estimated using an internal model which also involves quite a bit of signal processing. This repo is based on a digital twin that we actually use with hardware in the loop. During testing we quickly realised that the signals provided by sensors similar to this pr are very different from the real robot. For this reason we use a momentum-based observer with some filtering to estimate external wrenches. This way we get a much more realistic signal. Obviously this cannot be done using MJCF. We currently have a paper in review that motivates this approach and I would like to implement that in this repo as well.
Maybe it makes sense to offer different versions within a single repo. So there could be a "pure" MJCF model and more specialised variants (like for dm_control or a Python reference implementation etc.).
Looking forward to hearing your opinion on this.
P.S.: The DeepMind guys mentioned that they want to create a "curated" repo with robot models for MuJoCo. This issue might be of more general interest and we could also discuss it over there and get their input as well.
We currently have a paper in review that motivates this approach and I would like to implement that in this repo as well.
Oh nice it would be interesting to learn more about your paper.
Maybe it makes sense to offer different versions within a single repo. So there could be a "pure" MJCF model and more specialised variants (like for dm_control or a Python reference implementation etc.).
Yes I agree that there may be some benefit to including a PyMJCF script with some basic customisation capabilities. I think the following was a nice example that @kevinzakka linked in an above comment.
The DeepMind guys mentioned that they want to create a "curated" repo with robot models for MuJoCo. This issue might be of more general interest and we could also discuss it over there and get their input as well.
Having a single curated repo sounds very valuable, I'd be happy to help contributing towards this as I begin to invest more time into dm_control.
Thanks for the above response on this @JeanElsner, I am now currently travelling atm so my responses may be slower until Thursday of this week. Nonetheless I am invested in helping out with this where I can.