VRM support

[takahirox]

This PR is a draft PR for supporting VRM in Hubs. Still under the experiment and very beginning of the implementation but I want to share the progress and want the feedbacks.

What is VRM?

VRM is a 3D model format extending glTF, primarily designed for humanoid character avatar.

Features

• Looking like Anime/Game characters with cartoon material
• Physics engine independent secondary animation for hair, cloths, and others
• Copyright information meta data
• Standard bone sets (so we can apply the same skinning key frame animation)
• Standard morph sets
• First person information (e.g. which node camera should be bound to)
• There are lot of downloadable VRM models for example at https://hub.vroid.com/en/

Refer to https://vrm.dev/en/vrm/ for more details

We Hubs team often got VRM support requests from users especially who are keen to VR and/or Animes/Games.

Screenshot

https://user-images.githubusercontent.com/7637832/122448785-fb4b7f00-cf59-11eb-8403-b969d87876b6.mp4

Changes

I'm working with the newer Three.js based on #4321 because older Three.js + VRM seems a bit problematic (in animation? according to @netpro2k)

For VRM specific feature handling, I'm trying three-vrm so far.

https://github.com/pixiv/three-vrm

If their API or behavior doesn't greatly fit to Hubs, we may need to send feature request to them, hack it, or write our own.

Importing VRM itself may not be so difficult (see the diff of gltf-model-plus.js). But we may need some works for animation and optimization stuffs as you see in the next section.

Note that the current implementation is very beginning of VRM support and some of them are hacky. There are lot of remaining tasks. Please see @TODO: in gltf-model-plus.js for the details.

Remaining tasks and challengings

More proper implement

• The secondary animation should be handled in the existing animation system or a new VRM animation system.

VR

• Map VR controller inputs to hands of VRM standard bone set.

bugs or limitations

• I succeeded in placing a VRM model but didn't do to in uploading as an avatar yet due to some errors.
• The secondary animation with three-vrm doesn't seem to work correctly if model's scale is not one. (If a model is placed it is resized.) We may need to hack it or send feature request.
• The secondary animation with three-vrm doesn't work for cloned VRM objects. We may need to create THREE.VRM objects for each cloned model.

Skinning model avatars

• networked-aframe needs to send skeleton transforms for avatars (unless each client applies an animation on its end)
• Support IK for Arms.
• Apply walking animation if VRM model avatars are moved.

Optimization

• Material optimization for low quality material preference. Maybe we can just use MeshBasicMaterial instead.
• Optimize raycasting. The FPS number significantly drops if I point the VRM model. I speculate it's because raycasting of high poly models. (Humanoid models are often high poly.)
• Evaluate Runtime and download time because humanoid models are often high poly. LOD may be a good optimization if they are slow.
• Evaluate the secondary animation performance. If it's slow, consider to load off from the main thread to workers.
• Consider that only the owner handles the secondary animation and sends transforms to other peers rather than handling it in all peers especially if it's slow
• Consider to load off IK from main thread to workers if we will start to support Arm IK

┆Issue is synchronized with this Jira Task

[albirrkarim]

cool

[Seikida]

https://twitter.com/Mikami_Gugenka/status/1561865773710069760?s=20&t=M0hp97ymyJ3XoZTo6t0sxA

[netpro2k]

We still want to add VRM support, but will be doing so as part of the new asset loading path (via BitECS). This branch will still be a useful reference (especially if we decide to use three-vrm) but closing the PR.