Blinking?

[skunkwerk]

Thanks for this great repo.

I tried the colab, but the resulting video doesn’t blink. Is that controllable from the pose file? I see there’s control over the eyes in the GUI, but not sure how to do it at inference time.

Thanks!

[ashawkey]

@skunkwerk Hi, yes, the pose file has an 'eye_ratio' entry for each frame here, but currently we don't use it in the inference code.

[skunkwerk]

Thanks.

I passed in the exp_eye parameter to test.py and tried to hard-code the area to 0.0 and also 0.15 in lines 161 & 163 in provider.py but neither caused the eyes to close. Am I doing something wrong?

If I can get that working, I think I can figure out how to add some random blinking by modulating that value over a few frames.

[ashawkey]

@skunkwerk Which dataset are you using? Since the blinking is learned from data too, some identities may fail to learn how to depend it on eye_ratio.

[skunkwerk]

Thanks. I was using the pretrained Obama model and generated this video with the eye area set to 0. I’ll try again with two other models and see if the issue still appears.

[ashawkey]

This is strange, the obama dataset should be able to control the eyes well as in the demo video. Maybe you could try with the GUI first?

[skunkwerk]

Thanks.

Setting the eye area to 0 works for the 'engm' model, but not for the others for some reason. But yes, I can see that in the demo video of the GUI the eye area does seem to work for the obama model. Which parameters does the GUI modify that causes that?

I haven't been able to try the GUI, as I'm running into this error on a fresh Ubuntu installation:

$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:07:16_PDT_2019 Cuda compilation tools, release 10.1, V10.1.243

$ nvidia-smi Sun Dec 25 20:25:38 2022
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro M4000 Off | 00000000:00:05.0 On | N/A |

$ python3 test.py --pose ~/Downloads/obama.json --ckpt ~/Downloads/obama_eo.pth --aud ~/Downloads/intro_eo.npy --workspace trial_obama -O --torso --bg_img ~/Downloads/bg.jpg --gui

$/RAD-NeRF/gridencoder/src/gridencoder.cu(309): error: no instance of overloaded function "atomicAdd" matches the argument list argument types are: (c10::impl::ScalarTypeToCPPTypeTc10::ScalarType::Double *, double) detected during: instantiation of "void kernel_grid_backward<scalar_t,D,C,N_C>(const scalar_t *, const float *, const scalar_t *, const int *, scalar_t *, uint32_t, uint32_t, float, uint32_t, uint32_t, __nv_bool) [with scalar_t=c10::impl::ScalarTypeToCPPTypeTc10::ScalarType::Double, D=5U, C=8U, N_C=2U]" (396): here instantiation of "void kernel_grid_backward_wrapper<scalar_t,D>(const scalar_t *, const float *, const scalar_t *, const int *, scalar_t *, uint32_t, uint32_t, uint32_t, float, uint32_t, scalar_t *, scalar_t *, uint32_t, __nv_bool) [with scalar_t=c10::impl::ScalarTypeToCPPTypeTc10::ScalarType::Double, D=5U]" (417): here instantiation of "void grid_encode_backward_cuda(const scalar_t *, const float *, const scalar_t *, const int *, scalar_t *, uint32_t, uint32_t, uint32_t, uint32_t, float, uint32_t, scalar_t *, scalar_t *, uint32_t, __nv_bool) [with scalar_t=c10::impl::ScalarTypeToCPPTypeTc10::ScalarType::Double]" (474): here

21 errors detected in the compilation of "/tmp/tmpxft_00001d2a_00000000-6_gridencoder.cpp1.ii". ninja: build stopped: subcommand failed.

[ashawkey]

It seems the GPU arch is too old to build the CUDA extension.