[MuJoCo Parser] Support for applying texture (for example from images) directly to primitive geometries

[agarwal-abhinav]

Is your feature request related to a problem? Please describe. Currently, the MuJoCo parser doesn't support applying texture directly to primitives. For instance, if a geometry is specified by primitives in the .xml file, and a texture is added to it via (say) a .png specified in the .xml, drake doesn't currently support loading that. This requires reasoning about converting the primitives to .obj, the texture mapping to .mtl, and then modifying the .xml to load the .obj. This is tedious pre-processing and doing it without human in the loop requires reasoning about u-v coordinate maps in a generic way, which isn't always convenient given multiple options for mapping texture in the .xml.

Describe the solution you'd like Currently, drake supports adding texture directly to primitive geometry in .sdf files for visualization in camera rendering.

1. First suggestion is to add the same feature for parsing mujoco .xml.
2. Second suggestion is to add the feature to visualize it meshcat as well. Doing this will not only allow rendering of the scene from camera, but also visualization of the robot trajectories.

Describe alternatives you've considered Even if the mapping to .obj works well for visualization in meshcat, it doesn't always work well in camera rendering (or vice-versa). Attached is an image as an example. Additionally, the right wall in the meshcat rendering can make it clear that getting u-v coordinates right isn't the most effortless pre-processing.

Additional context There is some elementary discussion about this on the drake developers slack. cc @SeanCurtis-TRI The attached example is part of an effort to load robocasa environments in drake for combining learning based robotics (needing a large library of feature rich yet varied environments) with model based tools like GCS. cc @GenericP3rson who has been working on this and provided the attached visualization. This issue is closely related to the work being done in #20444 cc @RussTedrake

[SeanCurtis-TRI]

This is related to #19076. When extracting material definitions from the mjcf file, they'll inevitably end up in GeometryProperties. However, those don't make it to Meshcat (as per the linked issue). So, the solution for this issue requires resolution of both issues.

[RussTedrake]

My understanding is that now that we have "in memory mesh files" support, a new potential solution to this problem is available, at least for the simple case (e.g. the anymal-c model, which is the first model missing textures called out in #20444):

1. the assets are already in obj
2. the texture elements in the mujoco xml just point to the respective pngs.

In this case, I think, we can just load the obj as an in-memory mesh file and add a trivial .mtl file + the .png to get things working. Note that the mujoco menagerie obj files (and many other mujoco files I've seen) actually include a reference to the .mtl file in the obj, but don't actually distribute that .mtl file -- it's simply missing. :see_no_evil:

FWIW - Here is the initial PR I had which implemented a lot of the texture parsing in the mujoco parser, but we decided it wasn't ready to land at that time due to the lack of support in Meshcat.

[agarwal-abhinav]

Adding some files to reproduce the exact issue we are facing when using robocasa.

A large number of objects in these mjcf files have no .obj associated with them. Rather, primitive geometries are created and assembled in the mujoco xml directly. The texture on these is also added directly in the mujoco xml, first by creating a texture, a corresponding material, and applying this material to each primitive directly.

A base case reproduction is mentioned in the attached zip file, which contains a mujoco xml where a room's walls are assembled in the xml using primitives and texture is added to them in the same file. Currently, they render as the default grey color in both meshcat and camera rendering engine (checked on VTK).

meshcat = StartMeshcat()

visualizer = ModelVisualizer(meshcat=meshcat)
ConfigureParser(visualizer.parser())
visualizer.AddModels('plain_wall.xml')
visualizer.Run(loop_once=0)

primitive_rendering_not_supported.zip

Even for SDF and URDF, adding texture via xml or code won't show up in meshcat. So for mujoco xml, we need to first make a fix in mujoco parser (which will help us in camera rendering), and the general request made in #19076 mentioned above (for meshcat).

Current method in the works is: creating an obj file of the same geometry as the primitive, manually adding uv coordinates similar to the way mujoco default texture application would, and creating an mtl file and specifying texture via the mtl file. This is similar to the solution mentioned by @RussTedrake in the comment above, except in this case we need to create both the .obj and the .mtl, including specifying u-v coordinates in the pre-processing step.

However, this solution results in the creation of a very large number of .obj files (since usually large number of primitives are assembled in the xml). This is thus not efficient, especially in cases where we want to use a large number of environments for robot learning applications for instance. Moreover, creating and manually specifying u-v coordinate maps becomes less trivial when large number of primitives are used to assemble objects (patterns of kitchen cabinets for instance).

cc @01110011011101010110010001101111 who is working on this.