multiview images inversion in one time

[ohjarwa]

Hi, thanks for your great work!

I slightly changed your code and use it to invert mulview images in one time. To be clearly, the inversion images are capture by different cameras at one time. The angle is from 0 degree(frontal) 30 45 60 90,both left and right. On the training time, I use the pre-compute camera labels to synthesis the specific view images by one w, and accumulate the loss of multiview together. However the result is failed. I wonder:

1. Is the pre-compute camera pose suitable for the mulview circumstance?
2. If not so, should I optimize the camera label during the step of w/w+ project or PTI .

Thanks.

[ohjarwa]

My description seems not clear. Here's a group of sample data.

[oneThousand1000]

1. Yes, the pre-compute camera pose is suitable, if you use the correct camera pose extraction code and image align code.

Could you please upload the aligned images of those images you showed? I wonder if the alignment is right.

My multi-view projection is a little different from yours. Specifically, all the multi-view images share a single w+ latent code (w latent code also works) and are rendered from different camera views. I didn't accumulate the loss of multiview together, instead, I backward the losses of different views respectively.

Here is part of my training code, it works for me.

        for i in tqdm(range(hyperparameters.max_pti_steps)):

            for pose_idx ,real_images_batch in enumerate(real_images_batchs):
                c = cameras[pose_idx]

                image_name = pose_idx
                generated_images = self.forward(w_pivot, c)['image'][0]

                loss, l2_loss_val, loss_lpips = self.calc_loss(generated_images, real_images_batch, image_name,
                                                               self.G, use_ball_holder, w_pivot, c)

                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                use_ball_holder = global_config.training_step % hyperparameters.locality_regularization_interval == 0

                global_config.training_step += 1

And here is a group of aligned images and camera parameters I use. Images are from FaceScape dataset: https://drive.google.com/file/d/1XESEBIUbCsOntoWrm3gj_vpoN1gNl9pg/view?usp=sharing

[ohjarwa]

Thanks for your reply.

I use the same w+ latent code too. I will check the img align results and compare the respective loss with accmulate loss. I'm not convenient now. Later I will upload the image align result.

Thank you. 😊