fp16 推理失败

[Alex611-ai]

Notice: In order to resolve issues more efficiently, please raise issue following the template. （注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

Code

#模型构建 funasr_model = AutoModel( model=get_config(FunAsrEnum.MODEL.value, "/data/fc-asr/model/iic/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch"), model_revision="v2.0.4", fp16=True, vad_model=get_config(FunAsrEnum.VAD_MODEL.value, "/data/fc-asr/model/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch"), vad_kwargs={"fp16": True}, punc_model=get_config(FunAsrEnum.PUNC_MODEL.value, "/data/fc-asr/model/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch"), punc_kwargs={"fp16": True}, spk_model=get_config(FunAsrEnum.SPK_MODEL.value, "/data/fc-asr/model/damo/speech_campplus_sv_zh-cn_16k-common"), spk_kwargs={"fp16": True}, batch_size=get_int_config(FunAsrEnum.BATCH_SIZE.value, 1), ncpu=get_int_config(FunAsrEnum.NUMBER_THREADS.value, 4), disable_pbar=True )

#推理 if sr != 16000: # resample with librosa data = librosa.resample(data, orig_sr=sr, target_sr=16000) if len(data.shape) == 2: # multi-channel wav input logger.warning("Input wav shape: {}, only first channel reserved.".format(data.shape)) data = data[:, 0] state['audio_input'] = (sr, data) rec_result = self.funasr_model.generate(data, return_spk_res=False, sentence_timestamp=True, return_raw_text=True, is_final=True, hotword=hotwords, output_dir=output_dir, pred_timestamp=self.lang == 'en', en_post_proc=self.lang == 'en', cache={})

#报错 File "F:\code\fc-asr\app\services\video_clip.py", line 65, in recog rec_result = self.funasr_model.generate(data, File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\auto\auto_model.py", line 317, in generate return self.inference_with_vad(input, input_len=input_len, **cfg) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\auto\auto_model.py", line 394, in inference_with_vad res = self.inference( File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\auto\auto_model.py", line 356, in inference res = model.inference(**batch, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\models\fsmn_vad_streaming\model.py", line 722, in inference segments_i = self.forward(**batch) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\models\fsmn_vad_streaming\model.py", line 562, in forward self.ComputeScores(feats, cache=cache) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\models\fsmn_vad_streaming\model.py", line 351, in ComputeScores scores = self.encoder(feats, cache=cache["encoder"]).to("cpu") # return B * T * D File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl return forward_call(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\models\fsmn_vad_streaming\encoder.py", line 260, in forward x1 = self.in_linear1(input) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl return forward_call(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\funasr\models\fsmn_vad_streaming\encoder.py", line 36, in forward output = self.linear(input) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl return forward_call(*args, **kwargs) File "D:\software\miniforge\envs\fc-asr\lib\site-packages\torch\nn\modules\linear.py", line 116, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

What have you tried?

What's your environment?

• OS (e.g., Linux):Windows
• FunASR Version (e.g., 1.0.0):1.2.4
• PyTorch Version (e.g., 2.0.0):2.3.1
• How you installed funasr (pip, source):pip
• Python version:3.11