Running AVSR model in colab causes kernel to restart

[nastia-lado]

Hi @wnhsu! I tried running the colab with AVSR model and there were two problems:

1. It throws an error as shown on the image (there was another person here who reported it) But this can be solved by commenting out the lines related to noise in the function
2. Kernel dies when running inference. It seems that it dies specifically on task.inference_step. It seems like it can be an issue with RAM. Several people reported it on fairseq github, but there is no reponse there

Will you be able to look into it?

[chevalierNoir]

@nastia-lado did you change anything inside the notebook? I just ran it and it works fine.

[nastia-lado]

Hi! @chevalierNoir! Thanks a lot for your reply. Yes, I changed the notebook because I tried to use audiovisual model

1. I tried downloading AV model instead of V presented in the original notebook !wget https://dl.fbaipublicfiles.com/avhubert/model/lrs3_vox/avsr/base_noise_pt_noise_ft_30h.pt
2. In function predict, the line with modalities changed: modalities = ["audio", "video"]
3. The line with prediction also changed - I've added audio file: hypo = predict(mouth_roi_path, audio_path, ckpt_path, user_dir)

[blevlabs]

@nastia-lado Did you ever get the audiovisual model to work? Trying to run it on my end.