Simplicits reproducibility of the 0.18.0 teaser demo

[rdecharette]

Dear authors,

thanks a lot for your great work -- it's truly really nice. I've been playing for some times with Simplicits, including with Gaussian Splats (following that tutorial). Everything works well but the precision of the solver is not as in kaolin 0.18.0 teaser video. In some configurations, the Gaussians literally explode (covariance increases drastically, objects gets highlighly deformed, etc.).

Can you elaborate on which parameters you recommend to replicate (or get closer to) the quality seen on the teaser:

• do you typically increase the qp points ? I tried with 10k qps for the doll, 500k for the tool scene.
• what is a reasonable number of newton steps ? I tried 3 (as in demo) and 10. The latter works better but not ideally.
• is timestep 0.03 sufficiently small for all deformation assumptions to hold (in particular, I noticed the gradient of NeoHookean material used the J-1 version of the first Piola)
• any other changes in physics solver params ?

Best.

Here is a visual after few simultation steps with 10k qps for the doll and 500k for the scene, scene.max_newton_steps = 3, scene.timestep = 0.03, scene.newton_hessian_regularizer = 1e-5. The doll has ym=2000. Noticed the extreme deformation of the doll (it doesn't always happen though in other initial conditions):

[itsvismay]

Hi,

I'm happy to help debug.

There is a high possibility that the training for the doll object did not converge within the default 10k steps. Could you try increasing the number of training steps in this line to 25k?

kaolin.physics.simplicits.SimplicitsObject.create_trained

[itsvismay]

If there is a lack of sufficient qps, it will result in jittery behavior in the undersampled regions; however 10k is more than enough. In fact, you could bring these down to the default 2k for the doll and 12-15k for the static scene

[rdecharette]

Hi @itsvismay,

thanks so much for your reply. I used 2k+ qps for doll and 100k for scene. I've been digging the problem over the week end but I'm not sure to really understand the problem.

I retrained a couple of times all models (doll, dozer). I noticed a few things:

• simulations of close-to-rigid object (yms > 1e6) works well.
• simulation of softer objects (yms in {2000, 21000, 200000}) is way less stable. Actually, model losses were quite high (with 2k sampling points, BS 10, num_handles 40) with le > 3000 and lo ~ 15000. For softer objects, I managed to lower it (~le < 500, lo ~10000) using more handles ~80. Is that the right way ?
• with the new training above, the simulation runs well (at least for 1 object) but the object penetrates the scene (eg, the doll falls all the way to the thigh into the table before being "expulsed"). I've been playing with collision parameters (larger radius, more penalty, etc.) but it doesn't work smoothly

By activating the debug loggin, I also noticed that newton solver often doesn't converge > is that expected ? I mostly see convergence after 1-10 iterations when objects are falling but as soon as one of them touches a "surface" it often does not converge anymore.

I've also been trying to do multiple object simulations (eg, 5-10 dolls). It works okay when very rigid but when you have a few of them softer it behaves weirdly (jittering, etc.). Any advices ?

thanks a lot for the help.

[rdecharette]

last question, I'm aiming to have a few (~10) objects with different physics props interacting in a scene. Do you believe it is doable with Kaolin or will be challenging ? Thanks.

[itsvismay]

Hi,

Happy to hear that the strange splat deformations were fixed with more training steps!

I'll try to answer all your questions below.

last question, I'm aiming to have a few (~10) objects with different physics props interacting in a scene. Do you believe it is doable with Kaolin or will be challenging ? Thanks.

It depends on the scene you're simulating. If it's 10 objects all getting pressed together in a box, this will be difficult to simulate. If its 10 objects scattered around the scene and maybe 4-5 in contact at any one time, this will be much easier and run much faster. Recently, I was able to simulate 4 fruits + a fruit bowl + background scene + robot gripper all in contact with each other using the sparse CG solver, with some initial contact parameter tuning.

simulation of softer objects (yms in {2000, 21000, 200000}) is way less stable. Actually, model losses were quite high (with 2k sampling points, BS 10, num_handles 40) with le > 3000 and lo ~ 15000. For softer objects, I managed to lower it (~le < 500, lo ~10000) using more handles ~80. Is that the right way ?

Currently, we do not scale the lo term (orthogonality loss) with the number of handles, so more handles will not help with model's convergence metrics since W'W - Id is not scaled with the number of handles. I recommend sticking to ~10-40 handles depending on the object's rigidness, batch size 10, and the final le~3000 and lo ~15000 is reasonable in this case with 40 handles. For stiffer objects, less handles are needed since they deform less.

Softer objects will, intuitively, produce lower elastic energies, but with 80 handles, you will potentially have convergence issues due to poor conditioning of the newton method hessian (ie, poorly conditioned mass matrix B' M B). Also, 80 handles for each object will result in huge Jacobians and could cause memory issues.

Lastly, we're working on improving the skinning weight discovery process to improve the speed of discovery, hessian conditioning, and orthogonality of skinning weights. In general, the heuristic I currently use for "training convergence" is:

1. le ~ <10k
2. lo decreases by a couple of orders of magnitude
3. lo ~ <10k (but depends on the number of handles)
4. More handles => more training steps

with the new training above, the simulation runs well (at least for 1 object) but the object penetrates the scene (eg, the doll falls all the way to the thigh into the table before being "expulsed"). I've been playing with collision parameters (larger radius, more penalty, etc.), but it doesn't work smoothly

This is very strange. Can you try the following steps to debug:

• As a sanity check, could you make sure scene.enable_collisions is called? Maybe collisions are turned off and the doll is bouncing on the floor?
• Can you make sure that the maximum number of collision pairs is set to ~10-15k per object in the scene? For two objects, around 30k pairs should be sufficient.
• Currently, the collision radius is globally constant, so large objects will require more qps (ex., for the doll scene, I set around 12-15k qps for the background and ~2k for the doll).
• Can you decrease the scene.timestep=0.01, increase the scene.max_newton_steps=20 to ensure better convergence?
• Can you ensure objects are not initialized to collide?

By activating the debug loggin, I also noticed that newton solver often doesn't converge > is that expected ? I mostly see convergence after 1-10 iterations when objects are falling but as soon as one of them touches a "surface" it often does not converge anymore.

This depends on the scene. As with other simulators, there's a tradeoff between convergence and speed. It's expected that the solver doesn't fully converge at every step when scene.max_newton_steps is small, or max_contact_pairs is too few. The torch.linalg.solve's direct solver works well for small scenes, but for larger scenes, sparse CG is required, and potentially a higher scene.newton_hessian_regularizer term is needed to help with convergence. Finally, you mentioned you've also been playing with the penalty stiffness for collisions. Having an extremely high penalty will cause numerical issues in the optimizer as well.

TLDR, if the sim's behavior is reasonable, I would not worry about convergence. If you notice issues, then I would:

1. Increase max_contact_pairs.
2. Increase scene.newton_hessian_regularizer (as a hack to force better conditioning)
3. Increase scene.max_newton_steps

[itsvismay]

I've also been trying to do multiple object simulations (eg, 5-10 dolls). It works okay when very rigid but when you have a few of them softer it behaves weirdly (jittering, etc.). Any advices ?

Jittery collisions might be due to either convergence issues or the contact pairs jumping around between steps. Increasing the max number of pairs will help.

[rdecharette]

Hello @itsvismay,

thank you so much for the very detailed reply -- and sorry for my delay. All your information are very useful.

Things have been progressing on my side, and while not 100% perfect, simulations are now much more reliable and working (though still a few issues with soft objects).

It depends on the scene you're simulating. If it's 10 objects all getting pressed together in a box, this will be difficult to simulate. If its 10 objects scattered around the scene and maybe 4-5 in contact at any one time, this will be much easier and run much faster. Recently, I was able to simulate 4 fruits + a fruit bowl + background scene + robot gripper all in contact with each other using the sparse CG solver, with some initial contact parameter tuning.

Yes, that's closer to my application. Your bowl results are exciting, eager to see such results in your next demo/publications.

with the new training above, the simulation runs well (at least for 1 object) but the object penetrates the scene (eg, the doll falls all the way to the thigh into the table before being "expulsed"). I've been playing with collision parameters (larger radius, more penalty, etc.), but it doesn't work smoothly

This is very strange.

In addition to retraining objects, I discovered a glitch in my code since I had accidently changed the collision_particle_radius. It was way to low, and plotting the scene points at different time steps revealed that pts where penetrating in between the particle radii leading to energy accumulation when finally particles collided and explosion of the object.
It now works much better -- although a side effect of increasing particule_radius is that object do not precisely contacts with the scene.

Thanks for sharing your training heuristics, that's super useful. I'll try scaling handles with softness as your advised.

This depends on the scene. As with other simulators, there's a tradeoff between convergence and speed. It's expected that the solver doesn't fully converge at every step when scene.max_newton_steps is small, or max_contact_pairs is too few. The torch.linalg.solve's direct solver works well for small scenes, but for larger scenes, sparse CG is required, and potentially a higher scene.newton_hessian_regularizer term is needed to help with convergence. Finally, you mentioned you've also been playing with the penalty stiffness for collisions. Having an extremely high penalty will cause numerical issues in the optimizer as well.

I still have a few explosions of objects which (in theory should not happen) in particular mostly with soft objects. https://github.com/user-attachments/assets/0470d1dd-a323-48f1-bf2c-cf1c2e746fe8 (a note: my simplicits object is correct, collision_radius is now okay -- I plotted the scene QPs points --, and the number of contact pairs is way sufficient. The same object works in other situations.)

I use 15k contact pairs and 20 newton max steps, which sufficient in most cases although with more objects logs show that we hit the contact limit sometimes. For steps, most step resolve in 1-10 steps.

Side note, with tiny objects I had to lower the conv_tol because otherwise it converges on the 0th iteration leading to no motion.

Thanks for all your advices. So far I'm on good track to use Kaolin for our project. I may create a few PRs as I applied a few updates on simplicits and the easy_api.

[itsvismay]

Hi,

Things have been progressing on my side, and while not 100% perfect, simulations are now much more reliable and working (though still a few issues with soft objects).

Thats good to hear!

In addition to retraining objects, I discovered a glitch in my code since I had accidently changed the collision_particle_radius. It was way to low, and plotting the scene points at different time steps revealed that pts where penetrating in between the particle radii leading to energy accumulation when finally particles collided and explosion of the object.

That's a good catch. The only option at the moment is to increase the num_qp. Its on my TODO list to decouple cubature/quadrature points from collision points, so hopefully in the next version I will be able to get this done in addition to per-object collision radius and friction parameters.

I still have a few explosions of objects which (in theory should not happen) in particular mostly with soft objects. https://github.com/user-attachments/assets/0470d1dd-a323-48f1-bf2c-cf1c2e746fe8

This appears to be one of two potential issues. Either an insufficient number of qps, or the timestep is too large, and tunneling occurs. Our collision bounds only work to prevent tunneling if contact has been detected between particles. If dt is large enough, particles will tunnel through, get stuck and result in jittery motion as they try to extra themselves.

One option is smaller timestep, and another option is a larger collision detection radius. The easiest solution is a smaller timestep. This unfortunately freezes the object in the air for numerical reasons (gradients go to 0 with tiny dt and tiny masses) if the object is small/light enough.

You mentioned noticing this here: "Side note, with tiny objects I had to lower the conv_tol because otherwise it converges on the 0th iteration leading to no motion." You might need to increase the object's density or increase volume for this to work. Unfortunately, I don't have a better solution to this problem at the moment. Maybe in the future, we can support a way to switch to fp64.

Thanks for all your advices. So far I'm on good track to use Kaolin for our project. I may create a few PRs as I applied a few updates on simplicits and the easy_api.

Thats great news! I'm happy to help. PRs, suggestions, and bug reports are all helpful!

[rdecharette]

great, thanks for the feedback.

This appears to be one of two potential issues. Either an insufficient number of qps, or the timestep is too large, and tunneling occurs. Our collision bounds only work to prevent tunneling if contact has been detected between particles. If dt is large enough, particles will tunnel through, get stuck and result in jittery motion as they try to extra themselves.

Very useful insight. I'll see if this problem persists with smaller dt (it's not a qps problem, i plotted the qps with proper radius). I also don't want to increase the collision detection radius because of its side effect (memory, collision before contact, etc.).

You mentioned noticing this here: "Side note, with tiny objects I had to lower the conv_tol because otherwise it converges on the 0th iteration leading to no motion." You might need to increase the object's density or increase volume for this to work. Unfortunately, I don't have a better solution to this problem at the moment. Maybe in the future, we can support a way to switch to fp64.

Maybe you missed my mention but decreasing conv_tol to 1e-5 worked fine. As far as I understand, it's a problem of energy being too small for tiny object, leading to immediate convergence.

fp64 would be great indeed, although I managed to make it work with a few tweaks here and there.

[rdecharette]

So: reducing the timestep (I used 0.01) worked well -- at least in the one buggy example I tested. A note is that it required adjusting conv_tol too to avoid convergence at 0th newton step.
Ultimately, I set conv_tol as a function of timestep, which I found much easier.

[itsvismay]

Thats a good idea. I'm happy it worked :)

[github-actions[bot]]

Stale issue, please reopen if still relevant