[BUG] VITS微调时报错

[SnowFox4004]

在vits微调5个epoch后报错，重新开始微调立即报错 似乎是validation时的问题， 给embedding层传了一个float类型的tensor 命令是

python fish_speech/train.py --config-name vits_decoder_finetune

修改了yml中的数据集路径以及batch_size pytorch版本2.3.1 cuda12.1

以下是日志

Error executing job with overrides: [] [rank0]: Traceback (most recent call last): [rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 139, in [rank0]: main() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main [rank0]: _run_hydra( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra [rank0]: _run_app( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app [rank0]: run_and_report( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report [rank0]: raise ex [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report [rank0]: return func() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in [rank0]: lambda: hydra.run( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run [rank0]: _ = ret.return_value [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value [rank0]: raise self._return_value [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job [rank0]: ret.return_value = task_function(task_cfg) [rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 135, in main [rank0]: train(cfg) [rank0]: File "/home/snowfox/fish_audio/fish_speech/utils/utils.py", line 77, in wrap [rank0]: raise ex [rank0]: File "/home/snowfox/fish_audio/fish_speech/utils/utils.py", line 66, in wrap [rank0]: metric_dict, object_dict = task_func(cfg=cfg) [rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 108, in train [rank0]: trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 543, in fit [rank0]: call._call_and_handle_interrupt( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt [rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch [rank0]: return function(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 579, in _fit_impl [rank0]: self._run(model, ckpt_path=ckpt_path) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 986, in _run [rank0]: results = self._run_stage() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1030, in _run_stage [rank0]: self.fit_loop.run() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 205, in run [rank0]: self.advance() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 363, in advance [rank0]: self.epoch_loop.run(self._data_fetcher) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 141, in run [rank0]: self.on_advance_end(data_fetcher) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 295, in on_advance_end [rank0]: self.val_loop.run() [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator [rank0]: return loop_run(self, *args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 135, in run [rank0]: self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 396, in _evaluation_step [rank0]: output = call._call_strategy_hook(trainer, hook_name, *step_args) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 311, in _call_strategy_hook [rank0]: output = fn(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 410, in validation_step [rank0]: return self._forward_redirection(self.model, self.lightning_module, "validation_step", *args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 640, in call [rank0]: wrapper_output = wrapper_module(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1593, in forward [rank0]: else self._run_ddp_forward(*inputs, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1411, in _run_ddp_forward [rank0]: return self.module(*inputs, **kwargs) # type: ignore[index] [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 633, in wrapped_forward [rank0]: out = method(_args, **_kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/lit_module.py", line 266, in validation_step [rank0]: prior_audios = self.generator.infer( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 549, in infer [rank0]: return self.decode( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 583, in decode [rank0]: x, m_p, logs_p, y_mask = self.enc_p( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 78, in forward [rank0]: text = self.text_embedding(text).transpose(1, 2) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward [rank0]: return F.embedding( [rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/functional.py", line 2265, in embedding [rank0]: return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) [rank0]: RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding) Epoch 5: 0%| | 0/17 [00:05<?, ?it/s]

在torch库里调用embedding前打印了一下发现前几个正常，中间有一个是Float类型的数据但是小数位为0