weird output

[mohitm1994]

Hi, Thanks a lot for the support. I tried out the steps to generate output using the example images. I get a weird output, can you guide me where I might be going wrong?? Thanks a lot again

https://user-images.githubusercontent.com/51849557/193658892-f95e81f6-28a9-486d-88af-7ab7bd6adccb.mp4

[mohitm1994]

Loading networks from "./projector/PTI/checkpoints/model_BVGKQAAQILLI_00018_w_plus.pth"... Reloading Modules! generate out/00018_w_plus.mp4 Setting up PyTorch plugin "bias_act_plugin"... Done. Setting up PyTorch plugin "upfirdn2d_plugin"... Done. 100%|██████████| 120/120 [00:15<00:00, 7.57it/s]

python gen_videos_from_given_latent_code.py --outdir=out --trunc=0.7 --npy_path ./projector_out/00018_w_plus/00018_w_plus.npy --network=./projector/PTI/checkpoints/model_BVGKQAAQILLI_00018_w_plus.pth --sample_mult=2

[mohitm1994]

I tried it out with another image and the output seems similar. Would really appreciate some help @oneThousand1000

https://user-images.githubusercontent.com/51849557/193680161-829aa02f-a41c-47b2-ad0e-de8242a0e5f6.mp4

[oneThousand1000]

Hi, I have never seen this kind of output before... It looks really weird.

Could you please upload the ckpt and latent code to Google Drive and share them with me?

[oneThousand1000]

I also want to know whether you made modifications to the code.

[oneThousand1000]

And you can try to use the ckpt and the gen_samples.py to generate some random images, it can help us to clarify whether the problem happened in video generation or the ckpt itself.

[mohitm1994]

Hi, I figured out the issue was installations. Specific versions of torchvision, wandb and lpips made it work.

[oneThousand1000]

Hi, I figured out the issue was installations. Specific versions of torchvision, wandb and lpips made it work.

Good!

[eliorkfacetrom]

Hi! Can you please specify the versions of those packages? Tried to install using their conda env and still got this weird result.

[oneThousand1000]

Hi! Can you please specify the versions of those packages? Tried to install using their conda env and still got this weird result.

Sorry for the missing installation guidance! For you reference:

torch                        1.11.0+cu113
torchvision                  0.12.0+cu113
wandb                        0.12.18
lpips                        0.1.4

torch and torchvision are installed by:

pip install torch===1.11.0+cu113 torchvision=== 0.12.0+cu113  -f https://download.pytorch.org/whl/torch_stable.html

I am struggling for cvpr, after cvpr DDL, I will update the installation guidance.

[oneThousand1000]

Hi @eliorkfacetrom, please comment on this issue if the versions of those packages work for you, I will update them in the readme.

[povolann]

I dug a bit into this because I had to update PyTorch and there was no other option (I got a new GPU). But somehow the problem was in Scipy (I have created 2 envs, both have the same Scipy, just different PyTorch, so it is a bit weird, anyway).... Problem comes from this line: interp = scipy.interpolate.interp1d(x, y, kind=kind, axis=0) And because interp1d is considered legacy, so you have to replace it. Since in the original code we use kind='cubic', I used CubicSpline:

from scipy.interpolate import CubicSpline
    for yi in range(grid_h):
        row = []
        for xi in range(grid_w):
            x = np.arange(-num_keyframes * wraps, num_keyframes * (wraps + 1))
            y = np.tile(ws[yi][xi].cpu().numpy(), [wraps * 2 + 1, 1, 1]) # (5, 14, 512)
            interpolator = CubicSpline(x, y, axis=0)
            interp = interpolator(np.linspace(-num_keyframes * wraps, num_keyframes * (wraps + 1), num=1))
            row.append(interp)
        grid.append(row)

Hmm, there might be needed some additional changes, because now this interp is numpy array, so I call it just w = torch.from_numpy(interp).to(device) not like this. My images look normal now, let me know if it worked for you with new PyTorch. Right now I was using:

scipy                     1.7.1
torch                     2.2.2+cu121
torchvision               0.17.2+cu121
wandb                     0.16.6
lpips                     0.1.4