threestudio icon indicating copy to clipboard operation
threestudio copied to clipboard

Published 20 hours ago • verified publisher icon threestudio-project

Support the static part of Control4d (experimental)

Open DSaurus opened this issue 1 year ago • 3 comments

  • Add experimental configs (controlnet-normal/canny, instructpix2pix, control4d-multiviews, magic3d-coarse/fine-multiviews).
  • Add multiview dataset which supports loading multi-view images and OpenCV camera model of NeRFStudio.
  • Add ControlNet and Instructpix2pix as new guidances.
  • Add GanVolumeRenderer, which combines nerf volume renderer with a GAN.
  • Add rgb-latent-material which simultaneously outputs rgb and latent features.
  • Add solid-rgb-feature-background as the background of Control4D.
  • Add magic3d_multiviews system and control4d_multiviews systems. magic3d_multiviews system will be changed to instruct-nerf2nerf system in the future. Control4d_multiviews system can achieve static scene editing given multi-view images.
  • Add GAN network utils and LPIPS loss utils.
  • Update ReadMe.md.
  • Add controlnet_aux to the requirements.txt.

DSaurus avatar Jun 10 '23 15:06 DSaurus

Wow, I just looked at your Control4D project page, the results are amazing! Congratulations and thanks for sharing your code!

claforte avatar Jun 13 '23 18:06 claforte

Wow, I just looked at your Control4D project page, the results are amazing! Congratulations and thanks for sharing your code!

Hi! Thank you for your kind words. I believe ThreeStudio is an incredibly powerful and scalable framework. It effectively combines numerous impressive 3D generation methods in a concise manner. Currently, I am working on extending ThreeStudio's capabilities to include 3D reconstruction from multi-view images and 3D editing. I will soon share some results obtained using ThreeStudio. Feel free to provide me with any advice or suggestions.

DSaurus avatar Jun 15 '23 09:06 DSaurus

I have extended Threestudio to support 3d reconstruction from multi-view images. Here are the reconstruction results on the instructnerf2nerf dataset.

https://github.com/threestudio-project/threestudio/assets/24589363/287d2e51-80d1-42e8-b31f-b68767dc7621

Additionally, threestudio now supports the editing of 3D scenes based on instructnerf2nerf. Here are some results of 3d editing.

https://github.com/threestudio-project/threestudio/assets/24589363/a534d99f-ee8e-4110-98b4-b07ef4d73df4

https://github.com/threestudio-project/threestudio/assets/24589363/382f45ae-348e-4d19-a976-c596a1603ce7

https://github.com/threestudio-project/threestudio/assets/24589363/fb6cb8af-6968-4a5e-9242-f3e406068fa0

DSaurus avatar Jun 19 '23 16:06 DSaurus

Looks good enough! I'll merge this PR.

bennyguo avatar Jun 23 '23 18:06 bennyguo