drake icon indicating copy to clipboard operation
drake copied to clipboard

[MuJoCo parser] Support parsing models from mujoco menagerie

Open RussTedrake opened this issue 1 year ago • 3 comments

The MuJoCo XML parser loads and properly displays all of the models in the DeepMind Control Suite (or will after my upcoming set of PRs) except for dog.xml, which is an outlier with tendons and skin, etc. I've opened this issue to keep track of the (sad) state of our current MuJoCo parser in loading the menagerie assets.

Models (with references to errors/warning described just below)

  • [ ] agilty_cassie. non-uniform scale (error 1)
  • [x] aloha
  • [ ] anybotics_anymal_b. more than one orientation (error 2)
  • [ ] anybotics_anymal_c. Loads all but textures (warning 1)
  • [x] bitcraze_crazyflie_2
  • [x] boston_dynamics_spot
  • [x] fruitfly. good enough / won't fix (warning 2); also, the wings don't show up?
  • [x] franka_emika_panda
  • [x] franka_fr3
  • [ ] google_barkour_v0. requires stl2obj (warning 3)
  • [ ] google_barkour_vb. requires stl2obj (warning 3)
  • [ ] google_robot. All but textures (warning 1)
  • [ ] hello_robot_stretch. non-uniform scale (error 1)
  • [ ] kinova_gen3. requires stl2obj (warning 3)
  • [x] kuka_iiwa_14
  • [ ] realsense_d435i. size from mesh (error 4)
  • [x] rethink_robotics_sawyer
  • [ ] robotiq_2f85. requires stl2obj (warning 3)
  • [ ] robotis_op3. requires stl2obj (warning 3)
  • [x] shadow_hand/right_hand.xml
  • [ ] shadow_hand/left_hand.xml. non-uniform scale (error 1)
  • [ ] shadow_new_hand. requires stl2obj (warning 3)
  • [ ] skydio_x2. All but textures (warning 1)
  • [ ] trossen_vx300s. requires stl2obj (warning 3)
  • [ ] trossen_wx250s. requires stl2obj (warning 3)
  • [ ] ufactory_lite6. requires stl2obj (warning 3)
  • [ ] ufactory_xarm7. requires stl2obj (warning 3)
  • [ ] unitree_a1. all but textures (warning 1)
  • [ ] unitree_g1. requires stl2obj (warning 3)
  • [ ] unitree_go1. requires stl2obj (warning 3)
  • [x] unitree_go2
  • [ ] unitree_h1. requires stl2obj (warning 3)
  • [ ] unitree_z1. requires stl2obj (warning 3)
  • [x] universal_robots_ur10e
  • [x] universal_robots_ur5e
  • [ ] wonik_allegro/right_hand.xml. default_dissipation (error 5)
  • [ ] wonik_allegro/left_hand.xml. default_dissipation (error 5)

Errors:

  1. "mesh {} was defined with a non-uniform scale; but Drake currently only supports uniform scaling". The most common use is to mirror a mesh file across an axis (e.g. to use a single mesh for left/ring fingers).
    • Status: This should be relatively easy to support once we have in-memory meshes (#15263). An alternative would be to have a "make mujoco xml drake friendly" script which manually preprocesses the mesh files.
  2. "Element geom has more than one orientation attribute specified (perhaps through defaults). There must be no more than one instance of quat, axisangle, euler, xyaxes, or zaxis"
    • Status: Just need to look into this one.
  3. "The OBJ file contains multiple objects; only OBJs with a single object are supported."
  4. "The size attribute for capsule geom must have at least two elements". This is actually a deeper issue; the capsule needs to get its size from the mesh.
  5. "GetCombinedHuntCrossleyDissipation(): condition 'default_dissipation >= 0.0'"

Important warnings/lack of support:

  1. Failed to load texture / Unsupported <texture>
    • Status: Need to open an issue summarizing our current ideas. Certainly, this will get easier once we have in-memory meshes that support textures (#15263)
  2. Unsupported <tendon>.
    • Status: not a priority to support.
  3. stl instead of obj.
    • Status: the parser correctly prints a recommendation for stl2obj. Proper stl support is coming (see here and here). Alternatively we could run stl2obj in a "make mujoco xml drake friendly" script.

To reproduce, run e.g.

from pydrake.all import ModelVisualizer, StartMeshcat, PackageMap

visualizer = ModelVisualizer(meshcat=meshcat)
visualizer.parser().package_map().AddRemote(
    package_name="mujoco_menagerie",
    params=PackageMap.RemoteParams(
        # This repository doesn't have up-to-date tags/releases; the scary
        # hash in the url is the most recent commit sha at the time of my
        # writing.
        urls=[
            f"https://github.com/google-deepmind/mujoco_menagerie/archive/bf110a75a56e5bd146c7ed76965737c71d48425d.tar.gz"
        ],
        sha256=("2562301c38ac82b593b31e523004b94c263b20952f99d2bcbb9939fa5c6bebd2"),
        strip_prefix="mujoco_menagerie-bf110a75a56e5bd146c7ed76965737c71d48425d/",
    ),
)
visualizer.AddModels(
    url="package://mujoco_menagerie/unitree_a1/a1.xml"
)
visualizer.Run(loop_once=True)

RussTedrake avatar Oct 29 '23 00:10 RussTedrake

See this (TRI internal) slack discussion for discussion about the extent to which we want to support "the legacy, non-standard, MuJoCo texture projection".

RussTedrake avatar Jun 29 '24 12:06 RussTedrake

#21666 resolves some errors. I've just done a complete sweep through the menagerie and made a detailed list of errors/warnings above.

RussTedrake avatar Jul 04 '24 13:07 RussTedrake