zalo
1st party WASM bindings are now available for Alpha testing
https://github.com/google-deepmind/mujoco/commit/4086261714d7cfbc1745d4c6cb0aa2116df45312 https://github.com/google-deepmind/mujoco/issues/2585#issuecomment-3473495118
Looked around a bit; it seems like a solid start 💪
Do you think it's at the point where it can load custom models and textures from the virtual filesystem?
I'm glad that the genned header has my beloved typed_memory_views... that should make the incremental porting effort much easier while I wrap my head around all of the newly exposed public functions... it would be nice if the typescript kept the typing for the arrays...
Perhaps I can have Claude take a stab at doing the port first 😅
@MatiasManevi @okmatija @sebasnorena-lgtm @kbayes @yuvaltassa Thank you for putting this WASM build together! I spent a bit porting my demo suite to use your new official WASM bindings: https://raw.githack.com/zalo/mujoco_wasm/use-official-wasm/index.html
A few versions ago, it seems like the way textures were stored and mapped from materials changed and I haven't been able to parse them since... With the new binding, I've ported most of the logic over to use the new accessors... but I still can't seem to find a way to access the textures properly 😅
Would it be possible to make it easier to access textures and uvs from materials in the official bindings?
I think the only meaningful change is that mat_textid is now a matrix, one row vector per material
int* mat_texid; // indices of textures; -1: none (nmat x mjNTEXROLE)
So each material can have multiple textures associated with it. You can identify the textures by the nonnegative index value, and its role by its location, where each column corresponds to one of the allowed mjtTextureRole roles.
If that explanation doesn't entirely clear things up, can you write what you are trying that doesn't work?
cc @haroonq @okmatija @erez-tom
Haha, perfect timing — we just finished hammering together our own little WASM build of MuJoCo, and then the official one showed up 😅
Feels like we were still sanding the table when the real carpenters walked in.
Our repo mujoco-wasm-forge focuses on keeping a full C ABI in WASM, auto-exporting functions from headers, and syncing versioned builds (3.2.x–3.3.x) through a reproducible toolchain.
Really excited to see an official path forming — we’ll test compatibility later and can share notes if it helps the rollout.
I think the only meaningful change is that
mat_textidis now a matrix, one row vector per material
Aha! That fixed texture loading! However, it seems like there is one last regression in my demo suite related to a change in how the mesh normals and UVs are indexed(?)
If you click into the Cassie demo here: https://raw.githack.com/zalo/mujoco_wasm/use-official-wasm/index.html It looks like the model's UVs and Normals are messed up... This was true in my own custom MuJoCo 3.4.X bindings, so it seems like some engine change since MuJoCo 2.3.X...
When I print out the information, it looks like the normals are in the correct range, so I think they're the correct datatype... and the vertices appear to be forming correct/proper triangles, so those aren't corrupt... And this time, I've checked mjmodel.h to ensure there isn't any formatting change that I didn't expect... What could be going on @yuvaltassa ? Could it be something like the normals and UVs are now "All X's, then All Y's, then All Z's..."?
@lshdlut Your demo isn't public! https://github.com/lshdlut/mujoco-wasm-play
The only relevant commit I can think of is https://github.com/google-deepmind/mujoco/commit/9756ed0d80dd7f45a478de8ea8b70e49754291b5 , does that seem relevant to you? Perhaps @quagla can shed some light, he made the change.
We might need an MRE to diagnose this...
Could it be something like the normals and UVs are now "All X's, then All Y's, then All Z's..."?
It is worth mentioning that the documentation to explain these sort of things is something we are still working on. Our Javascript bindings expose underlying matrix in a very specific way.
When a function from the MuJoCo API returns a matrix (or needs a matrix as input), these are represented in the JavaScript side as flat, one-dimensional TypedArray arrays. The elements are stored in row-major order.
For example, a 3x10 matrix will be returned as a flat array with 30 elements. The first 10 elements represent the first row, the next 10 represent the second row, and so on.
Example: Accessing an element at (row, col)
// A 3x10 matrix is stored as a flat array.
const matrix: Float64Array = ...;
const nRows = 3;
const nCols = 10;
// To access the element at row `i` and column `j`:
const element = matrix[i * nCols + j];
The only relevant commit I can think of is google-deepmind/mujoco@9756ed0 , does that seem relevant to you? Perhaps @quagla can shed some light, he made the change.
Aha! This was it! I needed to use the mesh_facetexcoord and mesh_facenormal arrays as another layer of indirection to map UVs and Normals back to vertex indices (which is how three.js indexes meshes). https://github.com/zalo/mujoco_wasm/pull/30/commits/0b76374279207eb7d6356cd1db2c2660d65261b1
Since my integration is now at full feature parity using these "Alpha" WASM bindings, I've merged them to the main branch 😎 https://zalo.github.io/mujoco_wasm/
Perhaps we can get some of the new control policy demos coming out of these various places integrated as a fun new online demo... would something like this be the best candidate? https://github.com/mujocolab/g1_spinkick_example
If MuJoCo itself included an evaluator or some kind of purpose-built interface for .onnx policies, that would probably make it easier to integrate demos and deploy MPCs 😅
Likewise, you're welcome to use/pick apart everything in this repo for the official MuJoCo example (which I'm sure will evolve over time to support most researchers' needs 😄).
@zalo Wow! This looks like a perfect fit for a project I've been developing over the past few months: https://github.com/ttktjmt/muwanx
Although I haven’t implemented full ONNX model validation yet, it already supports online inference and execution!
In addition, the system can automatically collect mujoco assets and resolve file paths on the WASM's VFS, requiring only the root MuJoCo model as input. With this, visualization of many models, including those from mujoco_menagerie, is already working successfully!
I’m also currently collaborating with the MyoSuite team, working on integrating reinforcement learning models for musculoskeletal simulations.
I also plan to make it directly callable from mjlab, since mjlab already manages observation/action info in python. This would make it possible to upload trained models as live demos directly to GitHub Pages right after training.
Cool! That looks like a great usage for it! The latest version should fix the issues with normals and textures (like on the Cassie robot), and I’ve improved the visuals a little too 😄
Thank you :) There's a lot to learn from your code It'd be great if mujoco supports this "mujoco scene => three.js scene" renderer officially (or as a separate package?). I can also contribute.
The basic locomotion and balancing with single-leg policies are integrated for the G1 model thanks to @julien-blanchon, but it'd take some additional tasks to support something like g1_spinkick_example, since it uses reference motion as its observation, but it's definitely possible and is in my todo list!
Sweet!
The intent of this repo is to provide more of a boilerplate for engineers to make their own custom visualizations (like Kevin and the FACET guys have done). I just refactored a few things to make that more clear while using classic JS developer "best-practices"😄
We could probably amend the example app.ts to serve a similar purpose (with the ability to visualize more things, or as more of a reusable set of helper functions for interfacing with three.js).
As far as an automated MuJoCo->three.js pipeline goes... I could see a universe where the three.js viewer can open arbitrary subfolders (which the main browsers now support), or all the assets can be packed via Base64 into the .html with the WASM and Javascript so it's a single-file executable (I did something similar for https://emulatorjs.org/ to generate embeddable single-file .html "executables" that load a retroarch wasm emulator and a .rom to play a game in the browser).
As for running ONNX policies... I really like packing everything into the browser too (and if there's a standard for it, let's do it!), but I could also see a universe where the MuJoCo Visualizer runs in the browser as a persistent thing, and it can optionally hook into some local Python instance via Websockets to send observations and receive control signals 🤔 Browser visualization might be compelling for robots operating on remote machines and across operating systems...
That's actually the same approach I'm taking with my project, but with online policy control. Hopefully, it will be callable from python only (w/o JS/TS modification) in the end. I'm trying to distribute it as an npm package first, and then package a python wrapper on PyPI.
One of the main purposes of Muwanx is to enable the entire simulation to function as a static site (for hosting on github pages), but this would be challenging if the onnx/mujoco file sizes increase. So I totally agree with your idea of having an option to visualise with or without servers.
Perhaps we should work together to avoid reinventing the wheel?
This is a very interesting thread. It would be great to figure out what feature set is useful for us to put into the DeepMind repo and what would be better as community projects. Perhaps the DeepMind repo should just focus on providing/maintaining bindings for the API and then we point to community projects for the higher-level components needed to make proper applications. We can discuss with @yuvaltassa.
It'd be great if mujoco supports this "mujoco scene => three.js scene" renderer officially (or as a separate package?).
We could probably amend the example app.ts to serve a similar purpose (with the ability to visualize more things, or as more of a reusable set of helper functions for interfacing with three.js).
The intent of this repo is to provide more of a boilerplate for engineers to make their own custom visualizations
I haven't looked at the code in a lot of detail but it is clearly much further along than the demo app we provided. We actually have an internal project which is better than the demo but even so I think a THREE.js renderer may be something that's better as a community maintained thing. The plan @zalo mentioned about providing boilerplate for making custom visualizations is pretty much exactly what we wanted the demo app to evolve into.
enable the entire simulation to function as a static site (for hosting on github pages), but this would be challenging if the onnx/mujoco file sizes increase.
This is something we are very interested in as well, @ttktjmt. Could you elaborate on what you mean by MuJoCo file size increasing? It would be great to host simulations on GitHub pages, that seems like a very convenient way for people to share their simulation code, right?
Our repo mujoco-wasm-forge focuses on keeping a full C ABI in WASM, auto-exporting functions from headers, and syncing versioned builds (3.2.x–3.3.x) through a reproducible toolchain.
This is a very interesting project, @lshdlut, it looks like your approach is to call C functions using ccall/cwrap. Does this accurately describe what you're doing? Can you elaborate on why you did things this way?
In the bindings we released we used embind, since it provided more readable user code and seemed to be fast enough (Zalo's original demo worked very well for example). That said, we (probably @MatiasManevi ) are planning to do some more comprehensive bench-marking sometime soon to compare native/x86 MuJoCo vs. WASM MuJoCo called via Embind. It would be very interesting if we could also add WASM MuJoCo called via ccall/cwrap using your work to this comparison.
I'm also curious about how large the WASM file you get is. @lshdlut, I assume it will be smaller than what we generate with Embind and that might be important for what @ttktjmt was talking about RE hosting simulations on Github pages. If its a lot smaller perhaps it would make sense to make it an option to generate bindings that way in the DeepMind repo. We are working hard on improving our Python scripts which generates the Embind bindings but it should be in a pretty good place in a couple of weeks and hopefully it will be easy to extent to incorporate your bindings :)
@okmatija Thanks for your thoughts on this!
Regarding your question about mujoco file size, I was referring to the size of mjcf scene files and onnx policy files, not the WASM binary itself. While Emscripten's memory can now grow without a hard limit, loading time becomes a concern. For example, a large scene like MyoSuite's Bimanual model takes around 20 seconds just to load the scene. Adding a policy on top of that increases the wait time further, preventing a smooth UX.
I agree with keeping the mujoco repo focused on maintaining clean API bindings, but we could maybe develop a package for converting mujoco scenes into a three.js scenes in a separate repo.
Quick updates on my end:
- Fixed uvs and normals and supported cube textures like in Leap Hand Cube. We can maybe replace the official live demo and open-source it for the community.
- Supported a VR viewer
- Still working on the best JS API and codebase structure (any advice would be super helpful..!!)
I'm happy to collaborate or contribute to whatever direction makes the most sense for the ecosystem!
My principle expertise is mostly in three.js, so if the added "Lines of Code" are in-scope, I can offer to extend the existing app.ts with:
- The ability to upload MuJoCo scenes + folder hierarchies to the viewer (+50-75 LoC)
- More comprehensive scene visualization (custom meshes, textures, etc.) (+200 LoC (?))
- Beautification (Reflective Floor (+225 LoC), Motion blur, Global Illumination (+20 LoC) etc.)
It's easy for me to port these things, but I want to be conscientious of how complex the "single-file introductory example" is to newcomers.
To this end, I personally try to keep the overall program flow in an intelligible main.js file (which they hopefully won't ever have to leave to make their demo page) and the "advanced scene parsing trivia" in a separate utils file.
@okmatija @zalo
https://github.com/lshdlut/mujoco-wasm-forge https://github.com/lshdlut/mujoco-wasm-play
Q1 – Are you really exposing the full C ABI through a simple wrapper layer? Yes. Motivated by the official binding, we moved from an adhoc rule based export list to our current “A∩B” pipeline: we scan mujoco.h/mjspec.h for all MJAPI declarations, intersect that with llvm-nm output from libmujoco.a, and auto-generate the mjwf* wrappers plus the EXPORTED_FUNCTIONS list. This gets us very close to the full public C ABI, with only a few systematic exclusions (non-core prefixes, variadics without *_v versions). On the JS side we treat the build as a plain Emscripten module and call functions via wasmExports/ccall/cwrap instead of Embind.
Q2 – Why use a plain C-ABI flow instead of Embind? We wanted something simple, direct, and fully automated: a clean C ABI, auto-generated wrappers, and a repeatable build flow for every MuJoCo version. (And to be honest, we know less about Embind.) Embind is richer but also more complex; our approach keeps the JS layer minimal and lets downstream projects choose their own TS/TypedArray style. It’s also easier to customize: changing what gets exported or how wrappers behave is mostly about adjusting the generator inputs, not hand-editing C++ bindings.
Q3 – What are the WASM sizes?Uncompressed, for current forge tags: – 3.2.5: mujoco.wasm ~2.6 MB, mujoco.js ~0.28 MB – 3.3.7: mujoco.wasm ~2.8 MB, mujoco.js ~0.38 MB – 3.3.8-alpha: mujoco.wasm ~2.7 MB, mujoco.js ~0.19 MB
Q4 – How does the C-ABI build perform compared to Embind? Our earlier benchmarks (3.3.8-alpha, emsdk 4.0.10, Node 22) showed mj_step/mj_forward running at almost the same cost in both builds: ~0.5–0.7 µs per step once warm. The differences show up in micro-benchmarks: calling wrappers directly via wasmExports and batching C entrypoints gives the C-ABI build lower JS↔WASM overhead for very small functions, while Embind pays extra for marshaling and object wrappers. These numbers are early/rough, so we treat them as directional rather than final.
Q5 – How do multi-version support and reproducible builds work?
Forge’s CI builds 3.2.5, 3.3.7, and 3.3.8-alpha using the same toolchain (emsdk 4.0.10 + Node 20).
For each version we clone MuJoCo by tag/commit, regenerate ABI metadata, generate wrapper exports, and run smoke/regression/mesh tests.
We treat dist/
Q6 – What is mujoco-wasm-play?mujoco-wasm-play is a full frontend application built on top of mujoco-wasm-forge outputs.
It loads dist/
Cool! Hopefully, it's a useful alternative for the cases where the official bindings don't cut it. Typescript support would be nice too.
I'm interested to see what it'll look like when/if the official team gets OpenGL rendering of the main viewer working in WASM.
@lshdlut Thanks for the extremely interesting details! You've done a really great job :)
-
Please let us know if there's something we can do to make the benchmarking/profiling you're working on easier, this is something we are also very interested it and its super cool that you have already set up an automated way of comparing our embind bindings with your more direct bindings.
-
Your mujoco-wasm-play application is amazing, @yuvaltassa you should check it out here (the link is from the repo README).
-
The code snippet you have on the forge repo is pretty nice actually, if you auto-generated the part where you assign variables to the function pointers (e.g.,
const compile = Module.cwrap('mjwf_mj_compile', 'number', ['number', 'number']);) then you would have something pretty ergonomic, right? It would be as convenient as the embind bindings actually. I really like how explicit the memory management stuff is too, that comes up in Embind too of course (e.g., typed_memory_view) but its just more hidden. -
Your point that "it’s also easier to customize: changing what gets exported or how wrappers behave is mostly about adjusting the generator inputs, not hand-editing C++ bindings" is a very good point! There has been quite a lot of engineering effort put into hand-editing the C++ bindings (mostly by collaborators like @MatiasManevi). I wonder if, in combination with some conveniences like the one mentioned above, we could end up in a better place with the official bindings by also removing embind...