magic-animate
magic-animate copied to clipboard
magic-research
Can we use multiple controlnet models
Hello,
I'm currently using the animation script as described in your README. I'm exploring the possibility of integrating both DensePose and OpenPose models within this pipeline. The reason behind this is that, while DensePose offers better pose accuracy, OpenPose tends to preserve facial features more effectively.
Could you advise on whether it's feasible to use both models concurrently in the current setup? If not, are there any recommended approaches or plans to implement this capability in future versions?
It appears that you have defined one controlnet in the class magicanimate.pipelines.pipeline_animation.AnimationPipeline. I am not sure if it is feasible to define another controlnet with the necessary dimensions.
Any guidance or suggestions on achieving this dual-model integration would be greatly appreciated.
Hi, thanks for your interest. I think it should be very fast to incorporate multiple controlnet without too many modifications. But we haven't tried yet, and there is no hands-on instructions at current stage.
@CenekSanzak Hi, have you tried to implement your idea of multicontrolnet?
@CenekSanzak Hi, have you tried to implement your idea of multicontrolnet?
No. Have your tried it? Or do you have a resourse for that? @quocanh34
@CenekSanzak I got some and I am trying to implement this. How can I dm you, telegram, facebook, ... for more discussions?
I got my raw version of adding multicontrolnet here (densepose+openpose): https://github.com/quocanh34/magic-animate-modified
The results are better in face. But I am trying to do face and hand only, because the skeleton may affect the results a lot.
Still a lot of hard coding but it is okay to have a look. (btw leave a star if you can reproduce it =__)))
@quocanh34 I saw your linkedin in your profile and I sent a connection request, I don't want to share my telegram publicly. Thank you for sharing the code, I will try to reproduce the results and inform you.