Fine-tuned on multiple input images

[rohitpaul23]

What is the proper way of fine-tuning the model on multiple images?

I passed three different images(.png files) of the same person and its corresponding poses(_densepose.npy)

It seems the model gets overfitted by the poses of the input images. So at the time of inference, the resultant frame is affected by the training image pose, even though the poses I passed at the time of inference don't contain any such poses.

When fined tuned the model with num_trained_epochs = 500 using 3 images, I am getting checkpoints from different epochs 166, 333, 499. Even though the results are better for epoch 166, for epochs 333 and 499 there are multiple reflections of the person images in a frame which somewhat correspond to the images used when finetuning

*The frame I am sharing shows the dual reflection of the person and the pose showing is not present in the pose_folder and is one of the poses supplied at the time of finetuning.

Thank You

[nikkozzblu]

I had a similar issue and came to the conclusion there was a typo in the code: https://github.com/johannakarras/DreamPose/blob/5bf30b7df70cf6f2e0bb25556c6ff2cbf0f2b1bf/finetune-unet.py#L191 should be frame_j = [example["frame_j"] for example in examples]

Working much better after that on multi-frame fine-tuning