How to move a bone?

[josh-burton]

The docs seem to suggest that moving a bone is possible:

https://developers.google.com/ar/develop/java/sceneform/animation/

Bones are movable by animation or accessed as a node via the SkeletonNode class. When bones are accessed as a Node, the new positions are overwritten every frame by animation if one is playing.

But I can't seem to find any examples of how. I have a model I'm importing I want to manually animate by moving the bones.

[alexey-pelykh]

I'm afraid it's not, we're struggling with the same. Computation of bone transforms from animation happen in native code, see com.google.ar.sceneform.animation.ModelSkeletonRig#updateBoneTransforms

Writing to boneTransformsBuffer does nothing, as it's not affecting anything. I guess it's a output-only buffer used by updateBoneTransformsNative

[alexey-pelykh]

Actually, I've found a way. You'll need some reflection to get skeletonRig from ModelRenderable, then use following code snipped for inspiration:

val modelSkeletonRig = (node.renderable as ModelRenderable).skeletonRig as ModelSkeletonRig
val boneIndex = modelSkeletonRig.getMaterialBoneIndex(materialBoneIndex)
val boneName = modelSkeletonRig.getBoneName(boneIndex)
modelSkeletonRig.materialBoneTransformsBuffer.position(materialBoneIndex shl 4)
modelSkeletonRig.materialBoneTransformsBuffer.put(matrix.data)

[GuillaumeBourge]

Actually, I've found a way. You'll need some reflection to get skeletonRig from ModelRenderable, then use following code snipped for inspiration:

val modelSkeletonRig = (node.renderable as ModelRenderable).skeletonRig as ModelSkeletonRig
val boneIndex = modelSkeletonRig.getMaterialBoneIndex(materialBoneIndex)
val boneName = modelSkeletonRig.getBoneName(boneIndex)
modelSkeletonRig.materialBoneTransformsBuffer.position(materialBoneIndex shl 4)
modelSkeletonRig.materialBoneTransformsBuffer.put(matrix.data)

Hello Alexey, can you post the part that allow you to get the skeletonRig? I tryed reflection on the skeletonRig variable of the ModelRenderable but i get an error that the object obtained by reflection cannot be cast into SkeletonRig... (same as ModelSkeletonRig)..

var skeletonRigObj = modelRenderable::class.java.getDeclaredField("skeletonRig")
skeletonRigObj.isAccessible = true
var skeletonRig = skeletonRigObj as SkeletonRig

I have to play partially animation of the model such as only play 35% of the animation and also play the animation in reverse..

[alexey-pelykh]

var method = ModelRenderable::class.java.getDeclaredMethod("getSkeletonRig")
method.isAccessible = true
return method.invoke(instanceOfModelRenderable) as? SkeletonRig

[alexey-pelykh]

@GuillaumeBo you should definitely look up reference how to work with reflection

[GuillaumeBourge]

Thanks you for the code, i had the exact same one (i already used reflexion on other application) I get

java.lang.ClassCastException: java.lang.reflect.Field cannot be cast to com.google.ar.sceneform.rendering.SkeletonRig when a try to play this code...

[alexey-pelykh]

You were casting field, not value of that field. Compare your and mine code line by line

[GuillaumeBourge]

Yes for the code i posted, but i tryed also to access the method "getSkeletonRig" before posting here.

Anyway, i found the issue, i used "getMethod("getSkeletonRig")" instead of getDeclaredMethod("getSkeletonRig")"

[shiaro007-joker]

Hi guys, I'm also in need of a similar functionality. But in my case I want to scale a bone of the skeleton. is there a way to do so? Any help would be highly appreciated. Thanks

[alexey-pelykh]

The approach is the same I guess: use the matrix

[shiaro007-joker]

The approach is the same I guess: use the matrix

I'm sorry but to be honest I have no idea about what matrix you are talking about here. I'm pretty new to Arcore. From what I understand from your previous comments I'm able to get the skeletonRig using reflection but I'm not able to understand what's matrix.data and how it contains the scaling of the bone? Can you please explain a little more or point me to a resource to understand the same? Thanks!

[alexey-pelykh]

I guess that's related to general "how skeletal animation works", and answer is "affine transformation matrix contains scaling data pretty much the same way as it does translation and rotation". Yet if that's the same for Sceneform - have no idea, haven't tried

[shiaro007-joker]

Thanks @alexey-pelykh. I think I found the matrix that needs to be changed in order to scale the bone. But I'm getting an BufferOverflowException at a later stage which I'm not able to understand why. Following is my code that I'm trying:

var method = ModelRenderable::class.java.getDeclaredMethod("getSkeletonRig")
method.isAccessible = true
var modelSkeletonRig = method.invoke(model) as? SkeletonRig
val boneIndex = modelSkeletonRig!!.getMaterialBoneIndex(1)
val boneName = modelSkeletonRig?.getBoneName(boneIndex) // I'm able to get the bone name here so I'm sure reflection is working fine.
val mat = Matrix()
modelSkeletonRig.getMatrixForBone(boneIndex,mat) // this is what I think is gonna return the matrix that we have to alter.
 modelSkeletonRig.materialBoneTransformsBuffer.put(mat.data) 

As soon as I insert the last line materialBoneTransformsBuffer.put(mat.data) in my code and run the app, I get an exception not at this same line of code but at a later stage in the internal arcore code.

java.nio.BufferOverflowException at com.google.android.filament.RenderableManager.setBonesAsMatrices(RenderableManager.java:212)

Can anyone please suggest what I'm doing wrong here? Thanks!

[alexey-pelykh]

I don't see saving, setting, and restoring buffer position in your code at all

modelSkeletonRig.materialBoneTransformsBuffer.position(materialBoneIndex shl 4)

[shiaro007-joker]

I don't see saving, setting, and restoring buffer position in your code at all

modelSkeletonRig.materialBoneTransformsBuffer.position(materialBoneIndex shl 4)

I tried setting the position as well like u suggested

modelSkeletonRig.materialBoneTransformsBuffer.position(1 shl 4)
              modelSkeletonRig.materialBoneTransformsBuffer.put(mat.data)

But I'm still getting the BufferOverflowException

[alexey-pelykh]

1. Do you restore position afterwards?
2. Does that bone have animations in the asset? If no - trick won't work
3. Are you using materialBoneIndex for index or boneIndex?