Errors with running demo

[Lcococi]

Could not import SPECTRE. Make sure you pull the repository with submodules to enable SPECTRE. Traceback (most recent call last): File "/mnt/workspace/inferno/inferno/models/temporal/external/SpectrePreprocessor.py", line 16, in from spectre.src.spectre import SPECTRE ModuleNotFoundError: No module named 'spectre.src'

Could not import EmoSwinModule. SWIN models will not be available. Make sure you pull the repository with submodules to enable SWIN. Could not import EmoSwinModule. SWIN models will not be available. Make sure you pull the repository with submodules to enable SWIN. SWIN not found, will not be able to use SWIN models Looking for checkpoint in '/mnt/workspace/inferno/assets/TalkingHead/models/EMOTE_v2/checkpoints' Found 1 checkpoints

• /mnt/workspace/inferno/assets/TalkingHead/models/EMOTE_v2/checkpoints/last.ckpt Selecting checkpoint '/mnt/workspace/inferno/assets/TalkingHead/models/EMOTE_v2/checkpoints/last.ckpt' Loading checkpoint '/mnt/workspace/inferno/assets/TalkingHead/models/EMOTE_v2/checkpoints/last.ckpt' Some weights of the model checkpoint at ../../../face/wav2vec2-base-960h were not used when initializing Wav2Vec2ModelResampled: ['lm_head.weight', 'lm_head.bias']
• This IS expected if you are initializing Wav2Vec2ModelResampled from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing Wav2Vec2ModelResampled from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of Wav2Vec2ModelResampled were not initialized from the model checkpoint at ../../../face/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. /home/pai/envs/work38/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at ../aten/src/ATen/native/BinaryOps.cpp:467.) return torch.floor_divide(self, other) Looking for checkpoint in '/mnt/workspace/inferno/assets/MotionPrior/models/FLINTv2/checkpoints' Found 1 checkpoints
• /mnt/workspace/inferno/assets/MotionPrior/models/FLINTv2/checkpoints/model-epoch=0758-val/loss_total=0.113977119327.ckpt Selecting checkpoint '/mnt/workspace/inferno/assets/MotionPrior/models/FLINTv2/checkpoints/model-epoch=0758-val/loss_total=0.113977119327.ckpt' creating the FLAME Decoder /mnt/workspace/inferno/inferno/models/DecaFLAME.py:93: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). torch.tensor(lmk_embeddings['dynamic_lmk_faces_idx'], dtype=torch.long)) /mnt/workspace/inferno/inferno/models/DecaFLAME.py:95: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). torch.tensor(lmk_embeddings['dynamic_lmk_bary_coords'], dtype=self.dtype)) creating the FLAME Decoder /home/pai/envs/work38/lib/python3.8/site-packages/pytorch_lightning/core/saving.py:209: UserWarning: Found keys that are not in the model state dict but in the checkpoint: ['renderer.render.dense_faces', 'renderer.render.faces', 'renderer.render.raw_uvcoords', 'renderer.render.uvcoords', 'renderer.render.uvfaces', 'renderer.render.face_uvcoords', 'renderer.render.face_colors', 'renderer.render.constant_factor'] rank_zero_warn( unable to load materials from: template.mtl 0%| | 0/8 [00:00<?, ?it/s] Traceback (most recent call last): File "demos/demo_eval_talking_head_on_audio.py", line 172, in main() File "demos/demo_eval_talking_head_on_audio.py", line 153, in main eval_talking_head_on_audio( File "demos/demo_eval_talking_head_on_audio.py", line 80, in eval_talking_head_on_audio run_evalutation(talking_head, File "/mnt/workspace/inferno/inferno_apps/TalkingHead/evaluation/evaluation_functions.py", line 373, in run_evalutation batch = talking_head(batch) File "/home/pai/envs/work38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/mnt/workspace/inferno/inferno_apps/TalkingHead/evaluation/TalkingHeadWrapper.py", line 120, in forward sample = self.talking_head_model(sample) File "/home/pai/envs/work38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/mnt/workspace/inferno/inferno/models/talkinghead/TalkingHeadBase.py", line 526, in forward sample = self.forward_audio(sample, train=train, desired_output_length=desired_output_length, **kwargs) File "/mnt/workspace/inferno/inferno/models/talkinghead/TalkingHeadBase.py", line 234, in forward_audio return self.audio_model(sample, train=train, desired_output_length=desired_output_length, **kwargs) File "/home/pai/envs/work38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/mnt/workspace/inferno/inferno/models/temporal/AudioEncoders.py", line 236, in forward return self._forward(sample, train=train, desired_output_length=desired_output_length) File "/mnt/workspace/inferno/inferno/models/temporal/AudioEncoders.py", line 176, in _forward proc = self.input_processor(raw_audio, sampling_rate=sample["samplerate"][0], return_tensors="pt") File "/home/pai/envs/work38/lib/python3.8/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 117, in call return self.current_processor(*args, **kwargs) File "/home/pai/envs/work38/lib/python3.8/site-packages/transformers/models/wav2vec2/feature_extraction_wav2vec2.py", line 179, in call raw_speech = np.asarray(raw_speech, dtype=np.float32) File "/home/pai/envs/work38/lib/python3.8/site-packages/torch/_tensor.py", line 645, in array return self.numpy().astype(dtype, copy=False) TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

[ryutyke]

same problem... spectre and EmoSwinModule. maybe probably related to this. https://github.com/filby89/spectre

• I erased didn't seem to be used, then EGL Problem (Windows)

https://github.com/radekd91/inferno/issues/7

last link not work

[yujia-mt]

I successfully run “TalkingHead/demos/demo_eval_talking_head_on_audio.py" with EMOTE_v2. But the result was incorrect. Obvious Shaking! What's the problem?

https://github.com/radekd91/inferno/assets/138568445/21c73f0f-372c-41b8-8d83-540057b0857c

[cucdengjunli]

mark

[zhangliyun9120]

The results are messy bad... Could you @radekd91 please check this? thx so much.

https://github.com/radekd91/inferno/assets/23353228/f6ea9407-7483-45fb-b5ea-dba21615fd5e

[radekd91]

This looks like something is wrong with your mesh export. As if you were saving the vertices shuffled. Not quite sure why.

How did you arrive at this result exactly?

On Tue, Apr 30, 2024, 22:43 Liyun @.***> wrote:

The results are messy bad... Could you @radekd91 https://github.com/radekd91 please check this? thx so much.

https://github.com/radekd91/inferno/assets/23353228/f6ea9407-7483-45fb-b5ea-dba21615fd5e

— Reply to this email directly, view it on GitHub https://github.com/radekd91/inferno/issues/9#issuecomment-2088012287, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHPXHJ5J6YQR4TLT2CTZAB6IRAVCNFSM6AAAAABCWKKOUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYGAYTEMRYG4 . You are receiving this because you were mentioned.Message ID: @.***>

[radekd91]

I successfully run “TalkingHead/demos/demo_eval_talking_head_on_audio.py" with EMOTE_v2. But the result was incorrect. Obvious Shaking! What's the problem? pyrender_with_audio.mp4

@yujia-mt this is definitely wrong. How did you call the demo? It should look like this:

https://github.com/radekd91/inferno/assets/4113693/ff6dad29-002a-43af-b1d6-8ab470c8a27b

Can you share the std output? It should print somewhere what was loaded (which checkpoints). This looks like an unconverged model.

[yinfan-24]

I encountered the same error. It seems to occur during the final rendering and composition of the video. Importing the obj files from "M005_Happy_0/meshes" into Blender works fine.

[Liu-hyp]

The results are messy bad... Could you @radekd91 please check this? thx so much.

pyrender_with_audio.mp4

@zhangliyun9120 @yinfan-24 I encountered the same error and fix it. It seems like the predicted vertices need to be assigned to the deep copy object of head template.obj, then converting this new psbody.mesh.Mesh object to trimesh.Trimesh object and rendering it. pred_vertices = predicted_vertices[t].detach().cpu().view(-1,3).numpy() mesh = copy.deepcopy(template_obj_ps) mesh.v = pred_vertices trimesh_mesh = trimesh.Trimesh(vertices=mesh.v, faces=template.faces) pred_image = renderer.render(trimesh_mesh) The parameter type of the render function should be modified.

[2hiTee]

The results are messy bad... Could you @radekd91 please check this? thx so much. pyrender_with_audio.mp4

@zhangliyun9120 @yinfan-24 I encountered the same error and fix it. It seems like the predicted vertices need to be assigned to the deep copy object of head template.obj, then converting this new psbody.mesh.Mesh object to trimesh.Trimesh object and rendering it. pred_vertices = predicted_vertices[t].detach().cpu().view(-1,3).numpy() mesh = copy.deepcopy(template_obj_ps) mesh.v = pred_vertices trimesh_mesh = trimesh.Trimesh(vertices=mesh.v, faces=template.faces) pred_image = renderer.render(trimesh_mesh) The parameter type of the render function should be modified.

Hi, following your answer, I still have not solved it. Can you screenshot your code? Thanks for your help!