Issue with Unity and MuJoCo 3 loading elasticity plugin

[dblanm]

Hi,

I'm a PhD candidate trying to use MuJoCo 3.1.0 with Unity for simulating deformable objects.

I have followed the instructions here and managed to set-up MuJoCo properly. Now I am able to import MuJoCo scenes (I have tried importing humanoid and sotfbox and the model is loaded into Unity). I have tried to import the sample scene of the poncho.

However I get the following error when trying to do so:

IOException: Error loading the model: XML Error: unknown plugin 'mujoco.elasticity.shell'
Element 'plugin', line 24

Mujoco.MjEngineTool.LoadModelFromFile (System.String fileName) (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Tools/MjEngineTool.cs:50)
Mujoco.MjImporterWithAssets.ImportFile (System.String filePath) (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Editor/Importer/MjImporterWithAssets.cs:53)
Mujoco.MjImporterEditorWindow.Apply () (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

OS: Windows 11 Unity Version: Unity 2022.3.19f1

For setting up MuJoCo in Unity I did the following:

1. Create a Unity project
2. Download the release 3.1.0 of MuJoCo for Windows.
3. Paste the mujoco.dll into my project Assets folder.
4. Clone the source files of MuJoCo.
5. Import the MuJoCo package in Unity by clicking: Window->Package Manager-> Import from local disk

I have tried copying the file elasticity.dll from the MuJoCo release that is inside Mujoco/bin/mujoco_plugin into my project Assets folder but this doesn't solve the issue.

How can I use the latest elasticity plugin with Unity? I'm guessing @Balint-H should know about this?

As a note I still experience the same issue as discussed in #1171 that when trying to run the scene MuJoCo doesn't work:

NullReferenceException: Failed to create Mujoco runtime.
Mujoco.MjScene.StepScene () (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Components/MjScene.cs:345)
Mujoco.MjScene.FixedUpdate () (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Components/MjScene.cs:98)

Before that it prompts the same error as mentioned in #845

IOException: Error loading the model: XML Error: problem reading attribute 'range'
Element 'joint', line 1

Mujoco.MjVfs.LoadXML (System.String filename) (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Tools/MjVfs.cs:83)
Mujoco.MjEngineTool.LoadModelFromString (System.String contents) (at C:/Users/Documents/UnityHub/project/mujoco_src/unity/Runtime/Tools/MjEngineTool.cs:69)
Mujoco.MjScene.CompileScene (System.Xml.XmlDocument mjcf, System.Collections.Generic.IEnumerable`1[T] components) (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Components/MjScene.cs:179)
Mujoco.MjScene.CreateScene (System.Boolean skipCompile) (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Components/MjScene.cs:171)
Mujoco.MjScene.Start () (at C:/Users/Documents/UnityHub project/mujoco_src/unity/Runtime/Components/MjScene.cs:89)

Thanks in advance!

[Balint-H]

Hello, the Unity package hasn't really supported composite bodies and the elasticity plugin so far, but would be a very good addition.

@erez-tom have you loaded any of the mujoco extension plugin binaries like that in the past?

[dblanm]

Hi @Balint-H, I can see that you did a PR that would support this feature in the plugin. I'd be more than happy to test it. Could you advise on how to do it? Should I compile MuJoCo from source and then import to Unity as usual?

[Balint-H]

Heya,

It's not strictly necessary to build from source to test it. As it was originally merged when the plugin was at 3.1.3 state, you can use the plugin from the PR branch (https://github.com/Balint-H/mujoco/tree/feature/unity-plugins).

You can get the plugin from there and use the 3.1.3 binary with it.

I recommend cloning the repo, checking out that branch and referencing it as a local package from file (as you might want to fiddle with the package's code)

Unfortunately the plugins are only used during import at the moment, so runtime features of them are expected to be lost (e.g. callbacks they use). However it should allow you to load e.g. the poncho or soft cube scenes. (I'm working at the moment to fix the rendering of those objects, currently they are not skinned but reveal their underlying collision geometry)

I'd really appreciate your feedback and potential contributions on figuring out how best we can fully support plugin features.

Please checkout the relevanr files I modified to load the plugins during import. It's possible that simply loading plugins when constructing scenes will work, or that implementing a new MjComponent is required.

https://github.com/Balint-H/mujoco/commit/c65410a92d22758245bd5fca451e1b137619ee83

Create a folder called "Plugins" in the unity/Runtime" folder and place the plugin binaries there.

[dblanm]

Great, thanks for the detailed explanation. I'll comment after testing.

[Balint-H]

In the current implementation, we are checking in the "Plugins" folder that should be in the mujoco/unity/Runtime. (I believe you may need to create it. I left it empty on purpose and version control ignored it, I'll fix it with the next PR). Then, add the binaries from the 3.1.3 build (MuJoCo/bin/mujoco_plugin) into the source mujoco/unity/Runtime/Plugins folder.

I didn't want to include the binaries by default, to prevent them from being unnecessarily version controlled. In the future we should add similar binary fetching process as with the main mujoco binary.

So in summary:

• Don't copy the source folder to Unity
• Add mujoco/unity/package.json from the Unity package manager tool
• Add Plugins folder to mujoco/unity/Runtime
• Download 3.1.3 build of MuJoCo
• Copy plugins from the build into the Plugins folder.

[Balint-H]

Can you tell me what kind of scene are you trying to load? That way when I try implementing plugin loading I can target that specifically.

[dblanm]

My idea is to load the poncho and then add the VR controllers with a weld equality that enables to grasp specific points of the mesh

[Balint-H]

https://github.com/google-deepmind/mujoco/assets/41113387/1426e6c8-9f1a-47ea-ae8b-81f89892c731

Okay that could definitely work even with the current state of the partial plugin support. The only issue is the rendering, but I've been experimenting with mapping composite bodies to the skinned renderers of Unity, I'll have a go at it this weekend for the poncho scene. But in case you want to experiment as well, here is the current script I'm working with, written for the hammock scene:

using System;
using System.Collections;
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Text.RegularExpressions;
using Unity.Collections;
using UnityEngine;

namespace Mujoco {
[RequireComponent(typeof(SkinnedMeshRenderer))]
[ExecuteInEditMode]
public class MjCompositeMeshBuilder : MonoBehaviour {

  private SkinnedMeshRenderer _meshRenderer;

  protected void Awake() {
    _meshRenderer = GetComponent<SkinnedMeshRenderer>();
  }


  public void GenerateSkinnedMesh() {
    DisposeCurrentMesh();
    List<Vector3> vertices = new List<Vector3>();
    List<Transform> transforms = new List<Transform>();
    var compositeRowColInfo = GetCompositeRowColInfo(transform).ToList();
    int ySize = compositeRowColInfo.Max(info => info.Item2);
    int xSize = compositeRowColInfo.Max(info => info.Item3);

    foreach ((var child, _, _) in compositeRowColInfo
               .OrderBy(info => info.Item2)
               .ThenBy(info => info.Item3)) {
      vertices.Add(child.localPosition);
      transforms.Add(child);
    }
    var mesh = new Mesh();
    // Name this mesh to easily track resources in Unity analysis tools.
    mesh.name = $"Mujoco mesh for composite {name}";
    mesh.vertices = vertices.ToArray();
    int[] triangles = new int[xSize * ySize * 6];
    for (int ti = 0, vi = 0, y = 0; y < ySize; y++, vi++) {
      for (int x = 0; x < xSize; x++, ti += 6, vi++) {
        triangles[ti] = vi;
        triangles[ti + 3] = triangles[ti + 2] = vi + 1;
        triangles[ti + 4] = triangles[ti + 1] = vi + xSize + 1;
        triangles[ti + 5] = vi + xSize + 2;
      }
    }

    mesh.triangles = triangles.Reverse().ToArray();


    Vector2[] uv = new Vector2[vertices.Count];
    Vector4[] tangents = new Vector4[vertices.Count];
    Vector4 tangent = new Vector4(1f, 0f, 0f, -1f);
    for (int i = 0, y = 0; y <= ySize; y++) {
      for (int x = 0; x <= xSize; x++, i++) {
        tangents[i] = tangent;
        uv[i] = new Vector2((float)x / xSize, (float)y / ySize);
      }
    }
    mesh.uv = uv;

    mesh.RecalculateNormals();
    mesh.RecalculateTangents();

    byte[] bonesPerVertex = Enumerable.Repeat(1, vertices.Count)
      .Select(i => (byte)i).ToArray();
    BoneWeight1[] boneWeights = transforms
      .Select((t, i) => new BoneWeight1 { boneIndex = i, weight = 1 }).ToArray();

    mesh.SetBoneWeights(new NativeArray<byte>(bonesPerVertex, Allocator.Temp),
    new NativeArray<BoneWeight1>(boneWeights, Allocator.Temp));

    mesh.bindposes = transforms
      .Select(t => Matrix4x4.TRS(t.localPosition, t.localRotation, Vector3.one).inverse).ToArray();
    _meshRenderer.sharedMesh = mesh;
    _meshRenderer.bones = transforms.ToArray();

  }

  protected void OnDestroy() {
    DisposeCurrentMesh();
  }

  // Dynamically created meshes with no references are only disposed automatically on scene changes.
  // This prevents resource leaks in case the host environment doesn't reload scenes.
  private void DisposeCurrentMesh() {
    if (_meshRenderer.sharedMesh != null) {
#if UNITY_EDITOR
      DestroyImmediate(_meshRenderer.sharedMesh);
#else
      Destroy(_meshFilter.sharedMesh);
#endif
    }
  }

  IEnumerable<Transform> GetDirectChildren(Transform parent) {
    foreach (Transform child in parent) {
      yield return child;
    }
  }

  IEnumerable<(Transform, int, int)> GetCompositeRowColInfo(Transform parent) {
    foreach (Transform child in parent) {
      string childName = child.name;

      // Regex hardcoded for naming convention of composite grid
      string pattern = @"([A-Za-z]+)(\d+)_(\d+)";
      Match match = Regex.Match(childName, pattern);

      if (match.Success) {
        yield return (child, int.Parse(match.Groups[2].Value), int.Parse(match.Groups[3].Value));
      }
    }
  }

}
}

The Unity tutorials of Jasper Flick were useful for making it work (https://catlikecoding.com/unity/tutorials/procedural-grid/).