fish-speech
fish-speech copied to clipboard
fishaudio
FileNotFoundError: [Errno 2] No such file or directory: 'data_server/target/release/data_server'
Traceback (most recent call last): File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/route_utils.py", line 231, in call_process_api output = await app.get_blocks().process_api( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/blocks.py", line 1591, in process_api result = await self.call_function( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/utils.py", line 678, in wrapper response = f(*args, **kwargs) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/webui/manage.py", line 332, in train_process subprocess.Popen(["data_server/target/release/data_server", "--files", "data/quantized-dataset-ft.protos"]) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/subprocess.py", line 971, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'data_server/target/release/data_server'
在训练的时候遇到这个报错
我这边发现没有这个目录自己手动创建了，但是再次执行训练遇到了以下报错 Traceback (most recent call last): File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/route_utils.py", line 231, in call_process_api output = await app.get_blocks().process_api( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/blocks.py", line 1591, in process_api result = await self.call_function( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/gradio/utils.py", line 678, in wrapper response = f(*args, **kwargs) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/webui/manage.py", line 332, in train_process subprocess.Popen(["data_server/target/release/data_server", "--files", "data/quantized-dataset-ft.protos"]) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/subprocess.py", line 971, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) PermissionError: [Errno 13] Permission denied: 'data_server/target/release/data_server'
根据语雀文档上的提示data_server已经安装上去了但是遇到以下报错。 1it [00:00, 3.03it/s] [2024-03-27T07:07:20Z INFO data_server] Arguments: Args { files: ["data/quantized-dataset-ft.protos"], causal: false, address: "127.0.0.1:50051" } [2024-03-27T07:07:20Z INFO data_server] Loaded 1 groups [2024-03-27T07:07:20Z INFO data_server] Starting server at 127.0.0.1:50051 Error: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" })) [2024-03-27 15:07:23,138][main][INFO] - [rank: 0] Instantiating datamodule <fish_speech.datasets.text.TextDataModule> [2024-03-27 15:07:23,374][datasets][INFO] - PyTorch version 2.2.1+cu118 available. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. [2024-03-27 15:07:38,067][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. [2024-03-27 15:07:43,940][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True [2024-03-27 15:07:43,941][main][INFO] - [rank: 0] Instantiating model <fish_speech.models.text2semantic.TextToSemantic> [2024-03-27 15:07:44,036][fish_speech.utils.utils][ERROR] - [rank: 0] Traceback (most recent call last): File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/utils.py", line 644, in _locate obj = getattr(obj, part) AttributeError: module 'fish_speech.models' has no attribute 'text2semantic'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/utils.py", line 650, in _locate
obj = import_module(mod)
File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target target = _locate(target) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/utils.py", line 658, in _locate raise ImportError( ImportError: Error loading 'fish_speech.models.text2semantic.TextToSemantic': ImportError('/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops9_pad_enum4callERKNS_6TensorEN3c108ArrayRefINS5_6SymIntEEElNS5_8optionalIdEE')
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/utils/utils.py", line 68, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/train.py", line 47, in train model: LightningModule = hydra.utils.instantiate(cfg.model) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate return instantiate_node( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node target = _resolve_target(node.get(_Keys.TARGET), full_key) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target raise InstantiationException(msg) from e hydra.errors.InstantiationException: Error locating target 'fish_speech.models.text2semantic.TextToSemantic', set env var HYDRA_FULL_ERROR=1 to see chained exception. full_key: model [2024-03-27 15:07:44,038][fish_speech.utils.utils][INFO] - [rank: 0] Output dir: results/text2semantic_400m_finetune_spk Error executing job with overrides: [] Error locating target 'fish_speech.models.text2semantic.TextToSemantic', set env var HYDRA_FULL_ERROR=1 to see chained exception. full_key: model
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2024-03-27T07:07:20Z INFO data_server] Starting server at 127.0.0.1:50051
Error: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
这里显示50051的端口被占用，原因未知。你试着手动结束占用端口的进程然后进行训练，直到不出现错误为止。 如果还是不行，关闭data_server.
2024-03-27T07:07:20Z INFO data_server] Starting server at 127.0.0.1:50051 Error: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))这里显示50051的端口被占用，原因未知。你试着手动结束占用端口的进程然后进行训练，直到不出现错误为止。 如果还是不行，关闭data_server.
现在data_server没有被占用，执行后报了以下错误
1it [00:00, 3.12it/s]
[2024-03-27T08:05:18Z INFO data_server] Arguments: Args { files: ["data/quantized-dataset-ft.protos"], causal: false, address: "127.0.0.1:50051" }
[2024-03-27T08:05:18Z INFO data_server] Loaded 1 groups
[2024-03-27T08:05:18Z INFO data_server] Starting server at 127.0.0.1:50051
[2024-03-27 16:05:21,443][main][INFO] - [rank: 0] Instantiating datamodule <fish_speech.datasets.text.TextDataModule>
[2024-03-27 16:05:21,677][datasets][INFO] - PyTorch version 2.2.1+cu118 available.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-27 16:05:36,284][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-27 16:05:42,266][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True
[2024-03-27 16:05:42,267][main][INFO] - [rank: 0] Instantiating model <fish_speech.models.text2semantic.TextToSemantic>
[2024-03-27 16:05:44,574][main][INFO] - [rank: 0] Instantiating callbacks...
[2024-03-27 16:05:44,574][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelCheckpoint>
[2024-03-27 16:05:44,578][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelSummary>
[2024-03-27 16:05:44,578][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.LearningRateMonitor>
[2024-03-27 16:05:44,579][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <fish_speech.callbacks.GradNormMonitor>
[2024-03-27 16:05:44,580][main][INFO] - [rank: 0] Instantiating loggers...
[2024-03-27 16:05:44,580][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating logger <lightning.pytorch.loggers.tensorboard.TensorBoardLogger>
[2024-03-27 16:05:44,587][main][INFO] - [rank: 0] Instantiating trainer <lightning.pytorch.trainer.Trainer>
Trainer already configured with model summary callbacks: [<class 'lightning.pytorch.callbacks.model_summary.ModelSummary'>]. Skipping setting a default ModelSummary callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Trainer(limit_val_batches=1) was configured so 1 batch will be used.
[2024-03-27 16:05:44,745][main][INFO] - [rank: 0] Logging hyperparameters!
[2024-03-27 16:05:44,780][main][INFO] - [rank: 0] Starting training!
[2024-03-27 16:05:44,780][main][INFO] - [rank: 0] Resuming from checkpoint: checkpoints/text2semantic-400m-v0.3-4k.pth
[2024-03-27 16:05:44,780][main][INFO] - [rank: 0] Resuming weights only!
[2024-03-27 16:05:45,160][main][INFO] - [rank: 0] Error loading state dict: <All keys matched successfully>
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
distributed_backend=nccl All distributed processes registered. Starting with 1 processes
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [5]
| Name | Type | Params
0 | model | Transformer | 384 M 1 | model.embeddings | Embedding | 38.0 M 2 | model.layers | ModuleList | 308 M 3 | model.norm | RMSNorm | 1.0 K 4 | model.output | Linear | 38.0 M
384 M Trainable params 0 Non-trainable params 384 M Total params 1,537.085 Total estimated model params size (MB) Sanity Checking: | | 0/? [00:00<?, ?it/s][2024-03-27 16:05:48,720][fish_speech.utils.utils][ERROR] - [rank: 0] Traceback (most recent call last): File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/utils/utils.py", line 68, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/train.py", line 98, in train trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit call._call_and_handle_interrupt( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch return function(*args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run results = self._run_stage() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage self._run_sanity_check() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check val_loop.run() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator return loop_run(self, *args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run batch, batch_idx, dataloader_idx = next(data_fetcher) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next batch = super().next() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next batch = next(self.iterator) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next out = next(self._iterator) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next out = next(self.iterators[0]) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next data = self._next_data() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data return self._process_data(data) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data data.reraise() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/_utils.py", line 717, in reraise exception = self.exc_type(msg) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/grpc/_channel.py", line 384, in init with state.condition: AttributeError: 'str' object has no attribute 'condition' [2024-03-27 16:05:48,728][fish_speech.utils.utils][INFO] - [rank: 0] Output dir: results/text2semantic_400m_finetune_spk Error executing job with overrides: [] Traceback (most recent call last): File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/train.py", line 125, in main train(cfg) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/utils/utils.py", line 79, in wrap raise ex File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/utils/utils.py", line 68, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "/home/heike/hkdata/Code/wrl/fish-speech-main/fish_speech/train.py", line 98, in train trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit call._call_and_handle_interrupt( File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch return function(*args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run results = self._run_stage() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage self._run_sanity_check() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check val_loop.run() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator return loop_run(self, *args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run batch, batch_idx, dataloader_idx = next(data_fetcher) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next batch = super().next() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next batch = next(self.iterator) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next out = next(self._iterator) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next out = next(self.iterators[0]) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next data = self._next_data() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data return self._process_data(data) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data data.reraise() File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/torch/_utils.py", line 717, in reraise exception = self.exc_type(msg) File "/home/heike/anaconda3/envs/fish-speech/lib/python3.10/site-packages/grpc/_channel.py", line 384, in init with state.condition: AttributeError: 'str' object has no attribute 'condition'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
试着看看预处理文件夹下有没有这些文件，是否为空？
如果还不行试着关掉数据服务器:
在根目录/fish_speech/datasets/text.py的这里：
# 第 187 行
def __init__(
self,
server: str = "localhost:50051",
seed: int = 42,
phones_prob: float = 0.3,
repetition_prob: float = 0.0,
interactive_prob: float = 0.5,
max_length: int = 1024,
tokenizer: AutoTokenizer = None,
use_speaker: bool = True,
use_data_server: bool = False if sys.platform == "win32" else True, # <-----------
proto_files: list = ["data/quantized-dataset-ft.protos"],
causual: bool = True,
mix_text_phone_prob: float = 0.5,
):
改成：
use_data_server: bool = False
我这边重新从视频下方下载了整合包并在liunx系统上部署了,跟着视频简介的文档执行完了,但是遇到以下问题
1it [00:00, 4.88it/s]
[2024-03-29T08:25:54Z INFO data_server] Arguments: Args { files: ["data/quantized-dataset-ft.protos"], causal: false, address: "127.0.0.1:50051" }
[2024-03-29T08:25:54Z INFO data_server] Loaded 1 groups
[2024-03-29T08:25:54Z INFO data_server] Starting server at 127.0.0.1:50051
[2024-03-29 16:25:57,196][main][INFO] - [rank: 0] Instantiating datamodule <fish_speech.datasets.text.TextDataModule>
[2024-03-29 16:25:57,422][datasets][INFO] - PyTorch version 2.2.2+cu118 available.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-29 16:26:11,359][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-29 16:26:17,096][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=True
[2024-03-29 16:26:17,097][main][INFO] - [rank: 0] Instantiating model <fish_speech.models.text2semantic.TextToSemantic>
[2024-03-29 16:26:19,185][main][INFO] - [rank: 0] Instantiating callbacks...
[2024-03-29 16:26:19,186][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelCheckpoint>
[2024-03-29 16:26:19,190][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelSummary>
[2024-03-29 16:26:19,190][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.LearningRateMonitor>
[2024-03-29 16:26:19,191][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <fish_speech.callbacks.GradNormMonitor>
[2024-03-29 16:26:19,192][main][INFO] - [rank: 0] Instantiating loggers...
[2024-03-29 16:26:19,192][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating logger <lightning.pytorch.loggers.tensorboard.TensorBoardLogger>
[2024-03-29 16:26:19,201][main][INFO] - [rank: 0] Instantiating trainer <lightning.pytorch.trainer.Trainer>
Trainer already configured with model summary callbacks: [<class 'lightning.pytorch.callbacks.model_summary.ModelSummary'>]. Skipping setting a default ModelSummary callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Trainer(limit_val_batches=1) was configured so 1 batch will be used.
[2024-03-29 16:26:19,352][main][INFO] - [rank: 0] Logging hyperparameters!
[2024-03-29 16:26:19,385][main][INFO] - [rank: 0] Starting training!
[2024-03-29 16:26:19,386][main][INFO] - [rank: 0] Resuming from checkpoint: checkpoints/text2semantic-400m-v0.3-4k.pth
[2024-03-29 16:26:19,386][main][INFO] - [rank: 0] Resuming weights only!
[2024-03-29 16:26:19,757][main][INFO] - [rank: 0] Error loading state dict: <All keys matched successfully>
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
distributed_backend=nccl All distributed processes registered. Starting with 1 processes
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1]
| Name | Type | Params
0 | model | Transformer | 384 M 1 | model.embeddings | Embedding | 38.0 M 2 | model.layers | ModuleList | 308 M 3 | model.norm | RMSNorm | 1.0 K 4 | model.output | Linear | 38.0 M
384 M Trainable params
0 Non-trainable params
384 M Total params
1,537.085 Total estimated model params size (MB)
Sanity Checking: | | 0/? [00:00<?, ?it/s]/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argumenttonum_workers=255in theDataLoader` to improve performance.
[2024-03-29 16:26:20,995][fish_speech.utils.utils][ERROR] - [rank: 0]
Traceback (most recent call last):
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 68, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 99, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next
batch = super().next()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next
batch = next(self.iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next
out = next(self._iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next
out = next(self.iterators[0])
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next
data = self._next_data()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
data.append(next(self.dataset_iter))
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 265, in iter
yield self.augment()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 341, in augment
response = self.sample_data()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 294, in sample_data
return self.stub.SampleData(request)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/grpc/_channel.py", line 1176, in call
return _end_unary_response_blocking(state, call, False, None)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Failed to select a group"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-03-29T16:26:20.825135169+08:00", grpc_status:13, grpc_message:"Failed to select a group"}"
[2024-03-29 16:26:20,997][fish_speech.utils.utils][INFO] - [rank: 0] Output dir: results/text2semantic_400m_finetune_spk Error executing job with overrides: [] Traceback (most recent call last): File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 126, in main train(cfg) File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 79, in wrap raise ex File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 68, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 99, in train trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit call._call_and_handle_interrupt( File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch return function(*args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run results = self._run_stage() File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage self._run_sanity_check() File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check val_loop.run() File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator return loop_run(self, *args, **kwargs) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run batch, batch_idx, dataloader_idx = next(data_fetcher) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next batch = super().next() File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next batch = next(self.iterator) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next out = next(self._iterator) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next out = next(self.iterators[0]) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next data = self._next_data() File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch data.append(next(self.dataset_iter)) File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 265, in iter yield self.augment() File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 341, in augment response = self.sample_data() File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 294, in sample_data return self.stub.SampleData(request) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/grpc/_channel.py", line 1176, in call return _end_unary_response_blocking(state, call, False, None) File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking raise _InactiveRpcError(state) # pytype: disable=not-instantiable grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INTERNAL details = "Failed to select a group" debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-03-29T16:26:20.825135169+08:00", grpc_status:13, grpc_message:"Failed to select a group"}"
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
文件夹只有这两个文件。
@AnyaCoder
use_data_server: bool = False 这个值修改成False会触发 ValueError: Total of weights must be greater than zero 以下是日志。
1it [00:00, 4.41it/s]
[2024-03-29T08:38:51Z INFO data_server] Arguments: Args { files: ["data/quantized-dataset-ft.protos"], causal: false, address: "127.0.0.1:50051" }
[2024-03-29T08:38:51Z INFO data_server] Loaded 1 groups
[2024-03-29T08:38:51Z INFO data_server] Starting server at 127.0.0.1:50051
[2024-03-29 16:38:53,659][main][INFO] - [rank: 0] Instantiating datamodule <fish_speech.datasets.text.TextDataModule>
[2024-03-29 16:38:53,870][datasets][INFO] - PyTorch version 2.2.2+cu118 available.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-29 16:39:06,907][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=False
[2024-03-29 16:39:06,908][fish_speech.datasets.text][INFO] - [rank: 0] Read total 1 groups of data
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[2024-03-29 16:39:12,155][fish_speech.datasets.text][INFO] - [rank: 0] use_data_server=False
[2024-03-29 16:39:12,155][fish_speech.datasets.text][INFO] - [rank: 0] Read total 1 groups of data
[2024-03-29 16:39:12,156][main][INFO] - [rank: 0] Instantiating model <fish_speech.models.text2semantic.TextToSemantic>
[2024-03-29 16:39:14,133][main][INFO] - [rank: 0] Instantiating callbacks...
[2024-03-29 16:39:14,134][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelCheckpoint>
[2024-03-29 16:39:14,138][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.ModelSummary>
[2024-03-29 16:39:14,138][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <lightning.pytorch.callbacks.LearningRateMonitor>
[2024-03-29 16:39:14,139][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating callback <fish_speech.callbacks.GradNormMonitor>
[2024-03-29 16:39:14,140][main][INFO] - [rank: 0] Instantiating loggers...
[2024-03-29 16:39:14,140][fish_speech.utils.instantiators][INFO] - [rank: 0] Instantiating logger <lightning.pytorch.loggers.tensorboard.TensorBoardLogger>
[2024-03-29 16:39:14,147][main][INFO] - [rank: 0] Instantiating trainer <lightning.pytorch.trainer.Trainer>
Trainer already configured with model summary callbacks: [<class 'lightning.pytorch.callbacks.model_summary.ModelSummary'>]. Skipping setting a default ModelSummary callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Trainer(limit_val_batches=1) was configured so 1 batch will be used.
[2024-03-29 16:39:14,298][main][INFO] - [rank: 0] Logging hyperparameters!
[2024-03-29 16:39:14,330][main][INFO] - [rank: 0] Starting training!
[2024-03-29 16:39:14,331][main][INFO] - [rank: 0] Resuming from checkpoint: checkpoints/text2semantic-400m-v0.3-4k.pth
[2024-03-29 16:39:14,331][main][INFO] - [rank: 0] Resuming weights only!
[2024-03-29 16:39:14,716][main][INFO] - [rank: 0] Error loading state dict: <All keys matched successfully>
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
distributed_backend=nccl All distributed processes registered. Starting with 1 processes
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1]
| Name | Type | Params
0 | model | Transformer | 384 M 1 | model.embeddings | Embedding | 38.0 M 2 | model.layers | ModuleList | 308 M 3 | model.norm | RMSNorm | 1.0 K 4 | model.output | Linear | 38.0 M
384 M Trainable params
0 Non-trainable params
384 M Total params
1,537.085 Total estimated model params size (MB)
Sanity Checking: | | 0/? [00:00<?, ?it/s]/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argumenttonum_workers=255in theDataLoader` to improve performance.
[2024-03-29 16:39:15,993][fish_speech.utils.utils][ERROR] - [rank: 0]
Traceback (most recent call last):
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 68, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 99, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next
batch = super().next()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next
batch = next(self.iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next
out = next(self._iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next
out = next(self.iterators[0])
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next
data = self._next_data()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
data.append(next(self.dataset_iter))
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 266, in iter
yield self.augment()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 342, in augment
response = self.sample_data()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 298, in sample_data
group = random.choices(
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/random.py", line 535, in choices
raise ValueError('Total of weights must be greater than zero')
ValueError: Total of weights must be greater than zero
[2024-03-29 16:39:15,995][fish_speech.utils.utils][INFO] - [rank: 0] Output dir: results/text2semantic_400m_finetune_spk
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 126, in main
train(cfg)
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 79, in wrap
raise ex
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/utils/utils.py", line 68, in wrap
metric_dict, object_dict = task_func(cfg=cfg)
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/train.py", line 99, in train
trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in _run
results = self._run_stage()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1031, in _run_stage
self._run_sanity_check()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1060, in _run_sanity_check
val_loop.run()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator
return loop_run(self, *args, **kwargs)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 128, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 133, in next
batch = super().next()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 60, in next
batch = next(self.iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in next
out = next(self._iterator)
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 142, in next
out = next(self.iterators[0])
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next
data = self._next_data()
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
data.append(next(self.dataset_iter))
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 266, in iter
yield self.augment()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 342, in augment
response = self.sample_data()
File "/home/heike/hkdata/Code/wrl/fish-speech/fish_speech/datasets/text.py", line 298, in sample_data
group = random.choices(
File "/home/heike/anaconda3/envs/fish-speech-new/lib/python3.10/random.py", line 535, in choices
raise ValueError('Total of weights must be greater than zero')
ValueError: Total of weights must be greater than zero
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
我的意思是,
detect.list是否有内容？SuShang文件夹内的音频是否为.wav格式？- 是否包含
.lab文件？
是的，detect.list中有内容，且SuShang文件夹内的音频格式为wav，文件夹中包含.lab文件。