SizheAn
Better How to guide? Step by step readme or something.
I have installed everything on a fresh Linux Mint latest version as of today. All the requirements are met, but it stops there, it seems firstly like there is an issue with: "UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3)
Cannot even get to the part where it generates the head, let alone, where do I feed it the image I want to create a head from?
@Lustgard Currently we're missing the camera pose estimation stage which means you can't use any random input image. What I did for testing was to use Stable Diffusion + ControlNet to generate new people in the exact same pose as the example dataset images, so I could keep the camera values from the existing dataset.json. Someone wrote up the instructions on my Reddit post about it: https://www.reddit.com/r/StableDiffusion/comments/14h0pf4/comment/jpf5qea/
If you want to generate the 3D PLY mesh as well as the video, you can check the required changes and dependencies on my forked repo: https://github.com/hack-mans/PanoHead
Thanks! I was able to solve the wrong NumPy version with the help of ChatGPT. For others who might need help: "conda install numpy=1.22.3" Simple yes, but I'm no linux master yet, so baby steps atm.
I wonder if it's possible to infer a camera pose if we know the orientation of the face, using MediaPipe's face mesh pose estimation as an example.
https://developers.google.com/mediapipe/solutions/vision/face_landmarker/python
@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off
@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off
I have actually tried using Deep3DFaceRecon to estimate the pose, however, the result is a lot worse than what you can get by using the given pose (see below). I wonder if the authors could kindly provide a hint of what they used to estimate face poses?
given pose:
deep3d pose:
@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off
I have actually tried using Deep3DFaceRecon to estimate the pose, however, the result is a lot worse than what you can get by using the given pose (see below). I wonder if the authors could kindly provide a hint of what they used to estimate face poses?
given pose:
deep3d pose:
Short answer is we use different cropping, centering, and pose estimation script other than Deep3DFaceRecon. And since that was a company service, I won't be able to share it. In fact, I myself also don't have the access now. Now I'm trying to find alternatives to achieve similar results but cannot guarantee either... Will keep you guys posted.
@SizheAn , do you know if the estimation ws done using a single image? Or this is a case where multi-image is used , calculated the camera data and just one image was used?
I'm trying to figure out how it woudl be possible to get such precise data using only one photo.
@SizheAn , do you know if the estimation ws done using a single image? Or this is a case where multi-image is used , calculated the camera data and just one image was used?
I'm trying to figure out how it woudl be possible to get such precise data using only one photo.
Only single image. Pretty accurate if you can detect facial landmarks in the image. Our method is a combination of company's service + 3DDFA_V2 (https://github.com/cleardusk/3DDFA_V2). You can check their examples.
We update the scripts and example data for obtaining camera poses and cropping the images for PTI. See https://github.com/SizheAn/PanoHead/blob/main/3DDFA_V2_cropping/cropping_guide.md
This is fantastic @SizheAn , this was massively helpful. Not only did I manage to get it up and running with the directions you provided but I'm also now more familiar with 3DDFA_V2 which is a pretty awesome project :D