GPT-SoVITS
GPT-SoVITS copied to clipboard
Published
20 hours ago •
RVC-Boss
RVC-Boss
Need some help training in other languages?
Hi, I'd like to know if you have an interest in some help to train GPT-SoVITS in other languages. I have some high-quality datasets with hundreds of thousands of hours of studio-quality audio for Spanish and Portuguese. If so, I can help with that. I didn't find the scripts for training and audio preprocessing for training the model from scratch. Do you have any guidelines that I could use?
Well, G2P is all you need. Find a suitable phonetic symbol, and use G2P to convert your label to phones.
But what about the intonation and prosody? I'm not sure if the pre-trained model can handle accents from Latin languages, such as Portuguese or Spanish.
Em qui., 25 de abr. de 2024 às 11:00, XXXXRT666 @.***> escreveu:
Well, G2P is all you need. Find a suitable phonetic symbol, and use G2P to convert your label to phones.
— Reply to this email directly, view it on GitHub https://github.com/RVC-Boss/GPT-SoVITS/issues/1018#issuecomment-2077267678, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKE457V7GTCFJB67P3FO3WDY7ED6DAVCNFSM6AAAAABGYYZN2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGI3DONRXHA . You are receiving this because you authored the thread.Message ID: @.***>
-- "A fact is a simple statement that everyone believes. It is innocent, unless found guilty. A hypothesis is a novel suggestion that no one wants to believe." - Edward Teller
Paulo Vasconcellos Principal Data Scientist
-- Esta mensagem de e-mail é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações legalmente privilegiadas, comercialmente sensíveis, privadas e/ou confidenciais. Se você não for o(s) destinatário(s) indicado(s), considere-se desde já notificado de que qualquer uso, divulgação, distribuição ou cópia não autorizada desta mensagem de e-mail ou seu conteúdo é estritamente proibido. Se você não for o destinatário indicado, comunique imediatamente o remetente por e-mail de resposta e destrua todas as cópias desta mensagem de e-mail.
This email message is for the sole use of the intended recipient(s) and may contain legally privileged, commercially sensitive, private, and/or confidential information. If you are not the intended recipient(s), you are hereby notified that any unauthorized use, disclosure, distribution, or copying of this email message or its contents is strictly prohibited. If you are not the intended recipient, please immediately notify the sender by reply email and destroy all copies of this email message.
Maybe you can train it without pre-trained models, or just, train a pre-trained model with your large dataset. Have a look at this #67
And if you want to use the pre-trained models, I think have a few hours dataset would be better
Thank you for this. I have some high-quality datasets in PT and ES with thousands of hours of studio-quality audio. I've seen #67 and it gave more clarity about the pipeline, but some details are still missing for me, such as parameters, learning rate, and so on.
Em qui., 25 de abr. de 2024 às 11:21, XXXXRT666 @.***> escreveu:
And if you want to use the pre-trained models, I think have a few hours dataset would be better
— Reply to this email directly, view it on GitHub https://github.com/RVC-Boss/GPT-SoVITS/issues/1018#issuecomment-2077327331, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKE457QHMUGEVWUZOCSZ3OLY7EGODAVCNFSM6AAAAABGYYZN2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGMZDOMZTGE . You are receiving this because you authored the thread.Message ID: @.***>
-- "A fact is a simple statement that everyone believes. It is innocent, unless found guilty. A hypothesis is a novel suggestion that no one wants to believe." - Edward Teller
Paulo Vasconcellos Principal Data Scientist
-- Esta mensagem de e-mail é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações legalmente privilegiadas, comercialmente sensíveis, privadas e/ou confidenciais. Se você não for o(s) destinatário(s) indicado(s), considere-se desde já notificado de que qualquer uso, divulgação, distribuição ou cópia não autorizada desta mensagem de e-mail ou seu conteúdo é estritamente proibido. Se você não for o destinatário indicado, comunique imediatamente o remetente por e-mail de resposta e destrua todas as cópias desta mensagem de e-mail.
This email message is for the sole use of the intended recipient(s) and may contain legally privileged, commercially sensitive, private, and/or confidential information. If you are not the intended recipient(s), you are hereby notified that any unauthorized use, disclosure, distribution, or copying of this email message or its contents is strictly prohibited. If you are not the intended recipient, please immediately notify the sender by reply email and destroy all copies of this email message.
It's said that learning rate is 1, and to train a pre-trained model, there is one more step.111
I think it means that we first train sovits model and the use this sovits model to extract the semantics token, then train GPT
It's said that learning rate is 1, and to train a pre-trained model, there is one more step.111
I think it means that we first train sovits model and the use this sovits model to extract the semantics token, then train GPT
Yes, I'm thinking the same. @RVC-Boss can you confirm if the stage1 and preproc of stage 1 is basically so-vits preprocessing/training. And then, in the second stage, we use the so-vits model to extract the tokens to train GPT?
Best
It's said that learning rate is 1, and to train a pre-trained model, there is one more step.111
I think it means that we first train sovits model and the use this sovits model to extract the semantics token, then train GPT
Yes, I'm thinking the same. @RVC-Boss can you confirm if the stage1 and preproc of stage 1 is basically so-vits preprocessing/training. And then, in the second stage, we use the so-vits model to extract the tokens to train GPT?
Best
You can freeze the token encoder of stage2, use the existed pretrained base model to extract the tokens to train s1, and then fine tune the stage2 decoder using your other languages g2p.
Your can also train the whole s2 model (fine tune or train from scratch), and the s1 pretrained base model can't be used for tine tuning.
@RVC-Boss Do I need to seperate the speakers?
A1.wav|ZH|SPK_1|Some text
B2.wav|ZH|SPK_2|Some text also
This will train a base model and it will have the ability to speak the new languages, or features between speakers will not be shared?
Also, I've saw that the base model training require a additional S2 stage training. Is the training code available? What should we do to produce this S2 training. I would be very grateful if you could let me know how these steps go.
@RVC-Boss Do I need to seperate the speakers?
A1.wav|ZH|SPK_1|Some text B2.wav|ZH|SPK_2|Some text alsoThis will train a base model and it will have the ability to speak the new languages, or features between speakers will not be shared?
speaker信息只是为了兼容以前一些其他软件的.list数据集文件格式，GPT-SoVITS实际上没有读取、利用这个信息，只要确保同一个音频里只有一个人在说话就行。
Also, I've saw that the base model training require a additional S2 stage training. Is the training code available? What should we do to produce this S2 training. I would be very grateful if you could let me know how these steps go.
https://github.com/RVC-Boss/GPT-SoVITS/wiki wiki上我补了下从零训练和训练新语言的一些信息
Also, I've saw that the base model training require a additional S2 stage training. Is the training code available? What should we do to produce this S2 training. I would be very grateful if you could let me know how these steps go.
https://github.com/RVC-Boss/GPT-SoVITS/wiki
wiki上我补了下从零训练和训练新语言的一些信息
謝謝。試了試，但是在get semantic仍然有size mismatched的Error （修改symbols list後）. 我該如何正確訓練？
"You can freeze the token encoder of stage2, use the existed pretrained base model to extract the tokens to train s1"
Do you mean, freeze_quantizer inside s2.json? It's true by default...
Btw, I'm using webui for dataset prepare.
Traceback (most recent call last):
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\prepare_datasets\3-get-semantic.py", line 62, in <module>
vq_model.load_state_dict(
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for SynthesizerTrn:
size mismatch for enc_p.text_embedding.weight: copying a param with shape torch.Size([322, 192]) from checkpoint, the shape in current model is torch.Size([807, 192]).
Traceback (most recent call last):
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\prepare_datasets\3-get-semantic.py", line 62, in <module>
vq_model.load_state_dict(
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for SynthesizerTrn:
size mismatch for enc_p.text_embedding.weight: copying a param with shape torch.Size([322, 192]) from checkpoint, the shape in current model is torch.Size([807, 192]).
Can't finetune SOVITS as well
s2 training is now finished. But after I used trained sovits model to generate the semantic tokens, then finetune the GPT model, I got error:
C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>set CUDA_LAUNCH_BLOCKING=1 && runtime\python.exe GPT_SoVITS/s1_train.py --config_file "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\TEMP/tmp_s1.yaml"
Seed set to 1234
Using 16bit Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
<All keys matched successfully>
ckpt_path: None
[rank: 0] Seed set to 1234
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=gloo
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------
semantic_data_len: 123720
phoneme_data_len: 123656
item_name semantic_audio
0 Audio_1.wav 80 130 209 53 385 296 50 60 322 79 181 312 7 5...
1 Audio_1000.wav 22 0 349 363 360 18 85 66 360 18 199 199 140 2...
2 Audio_1005.wav 50 267 267 130 130 53 130 130 296 130 96 155 1...
3 Audio_1007.wav 322 69 69 17 298 223 99 135 1 233 4 9 551 433 ...
4 Audio_101.wav 22 130 322 64 119 963 299 433 278 144 500 272 ...
... ... ...
123715 f5e88e3aa01bfc3d315de53232a842787cda7d83#84.wav 249 195 195 195 249 22 22 22 22 22 85 143 143 ...
123716 f5e88e3aa01bfc3d315de53232a842787cda7d83#89.wav 249 195 195 143 143 22 22 22 22 22 22 22 22 85...
123717 f5e88e3aa01bfc3d315de53232a842787cda7d83#90.wav 249 195 143 143 143 55 143 143 143 143 248 444...
123718 f5e88e3aa01bfc3d315de53232a842787cda7d83#93.wav 249 444 527 775 359 18 135 662 249 775 125 206...
123719 f5e88e3aa01bfc3d315de53232a842787cda7d83#98.wav 249 195 195 143 143 250 22 22 22 22 22 195 143...
[123720 rows x 2 columns]
deleted 5 audios who's duration are bigger than 54 seconds
deleted 1912 audios who's phoneme/sec are bigger than 25 or smaller than 3
dataset.__len__(): 121803
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
-----------------------------------------------
0 | model | Text2SemanticDecoder | 77.5 M
-----------------------------------------------
77.5 M Trainable params
0 Non-trainable params
77.5 M Total params
309.975 Total estimated model params size (MB)
Epoch 0: 0%| | 0/15226 [00:00<?, ?it/s]C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\launchers\subprocess_script.py", line 102, in launch
return function(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 989, in _run
results = self._run_stage()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1035, in _run_stage
self.fit_loop.run()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 202, in run
self.advance()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\training_epoch_loop.py", line 136, in run
self.advance(data_fetcher)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\training_epoch_loop.py", line 242, in advance
batch_output = self.manual_optimization.run(kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\optimization\manual.py", line 92, in run
self.advance(kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\optimization\manual.py", line 112, in advance
training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 381, in training_step
return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 633, in __call__
wrapper_output = wrapper_module(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\parallel\distributed.py", line 1156, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0]) # type: ignore[index]
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 626, in wrapped_forward
out = method(*_args, **_kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\models\t2s_lightning_module.py", line 39, in training_step
loss, acc = forward(
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\models\t2s_model.py", line 190, in forward_old
x = self.ar_text_embedding(x)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\modules\embedding.py", line 31, in forward
x = self.word_embeddings(x)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\sparse.py", line 162, in forward
return F.embedding(
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\functional.py", line 2210, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 182, in <module>
main(args)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 158, in main
trainer.fit(model, data_module, ckpt_path=ckpt_path)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 68, in _call_and_handle_interrupt
trainer._teardown()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1012, in _teardown
self.strategy.teardown()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\ddp.py", line 406, in teardown
super().teardown()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\parallel.py", line 127, in teardown
super().teardown()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 528, in teardown
self.lightning_module.cpu()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 79, in cpu
return super().cpu()
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 954, in cpu
return self._apply(lambda t: t.cpu())
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
module._apply(fn)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
module._apply(fn)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
param_applied = fn(param)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 954, in <lambda>
return self._apply(lambda t: t.cpu())
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Epoch 0: 0%| | 0/15226 [00:00<?, ?it/s]
C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>
After set cpu as train device, I could see the main error:
try:
# 使用正则表达式匹配文件名中的数字部分，并按数字大小进行排序
newest_ckpt_name = get_newest_ckpt(os.listdir(ckpt_dir))
ckpt_path = ckpt_dir / newest_ckpt_name
except Exception:
ckpt_path = None
print("ckpt_path:", ckpt_path)
trainer.fit(model, data_module, ckpt_path=ckpt_path)
C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>set CUDA_LAUNCH_BLOCKING=1 && runtime\python.exe GPT_SoVITS/s1_train.py --config_file "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\TEMP/tmp_s1.yaml"
Seed set to 1234
Using 16bit Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
<All keys matched successfully>
Traceback (most recent call last):
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 186, in <module>
main(args)
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 159, in main
newest_ckpt_name = get_newest_ckpt(os.listdir(ckpt_dir))
File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\utils\__init__.py", line 24, in get_newest_ckpt
newest_ckpt = sorted_info[0][2]
IndexError: list index out of range
What I should do? Referring to the wiki I thought I could jump right in after adding the new language g2p, but I got much more than that.
I usually train Portuguese models using IPA, is it possible to use IPA to train the model? If so, can anyone can help me with the steps, I have around 660 hours of untranscribed data that I plan to transcribe using Whisper and a script to cut the audio.
s2 training is now finished. But after I used trained sovits model to generate the semantic tokens, then finetune the GPT model, I got error:
C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>set CUDA_LAUNCH_BLOCKING=1 && runtime\python.exe GPT_SoVITS/s1_train.py --config_file "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\TEMP/tmp_s1.yaml" Seed set to 1234 Using 16bit Automatic Mixed Precision (AMP) GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs <All keys matched successfully> ckpt_path: None [rank: 0] Seed set to 1234 Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1 ---------------------------------------------------------------------------------------------------- distributed_backend=gloo All distributed processes registered. Starting with 1 processes ---------------------------------------------------------------------------------------------------- semantic_data_len: 123720 phoneme_data_len: 123656 item_name semantic_audio 0 Audio_1.wav 80 130 209 53 385 296 50 60 322 79 181 312 7 5... 1 Audio_1000.wav 22 0 349 363 360 18 85 66 360 18 199 199 140 2... 2 Audio_1005.wav 50 267 267 130 130 53 130 130 296 130 96 155 1... 3 Audio_1007.wav 322 69 69 17 298 223 99 135 1 233 4 9 551 433 ... 4 Audio_101.wav 22 130 322 64 119 963 299 433 278 144 500 272 ... ... ... ... 123715 f5e88e3aa01bfc3d315de53232a842787cda7d83#84.wav 249 195 195 195 249 22 22 22 22 22 85 143 143 ... 123716 f5e88e3aa01bfc3d315de53232a842787cda7d83#89.wav 249 195 195 143 143 22 22 22 22 22 22 22 22 85... 123717 f5e88e3aa01bfc3d315de53232a842787cda7d83#90.wav 249 195 143 143 143 55 143 143 143 143 248 444... 123718 f5e88e3aa01bfc3d315de53232a842787cda7d83#93.wav 249 444 527 775 359 18 135 662 249 775 125 206... 123719 f5e88e3aa01bfc3d315de53232a842787cda7d83#98.wav 249 195 195 143 143 250 22 22 22 22 22 195 143... [123720 rows x 2 columns] deleted 5 audios who's duration are bigger than 54 seconds deleted 1912 audios who's phoneme/sec are bigger than 25 or smaller than 3 dataset.__len__(): 121803 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] | Name | Type | Params ----------------------------------------------- 0 | model | Text2SemanticDecoder | 77.5 M ----------------------------------------------- 77.5 M Trainable params 0 Non-trainable params 77.5 M Total params 309.975 Total estimated model params size (MB) Epoch 0: 0%| | 0/15226 [00:00<?, ?it/s]C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [60,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [165,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [160,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed. Traceback (most recent call last): File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\launchers\subprocess_script.py", line 102, in launch return function(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 580, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 989, in _run results = self._run_stage() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1035, in _run_stage self.fit_loop.run() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 202, in run self.advance() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 359, in advance self.epoch_loop.run(self._data_fetcher) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\training_epoch_loop.py", line 136, in run self.advance(data_fetcher) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\training_epoch_loop.py", line 242, in advance batch_output = self.manual_optimization.run(kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\optimization\manual.py", line 92, in run self.advance(kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\loops\optimization\manual.py", line 112, in advance training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values()) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 309, in _call_strategy_hook output = fn(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 381, in training_step return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 633, in __call__ wrapper_output = wrapper_module(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\parallel\distributed.py", line 1156, in forward output = self._run_ddp_forward(*inputs, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward return module_to_run(*inputs[0], **kwargs[0]) # type: ignore[index] File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 626, in wrapped_forward out = method(*_args, **_kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\models\t2s_lightning_module.py", line 39, in training_step loss, acc = forward( File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\models\t2s_model.py", line 190, in forward_old x = self.ar_text_embedding(x) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\modules\embedding.py", line 31, in forward x = self.word_embeddings(x) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\sparse.py", line 162, in forward return F.embedding( File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\functional.py", line 2210, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 182, in <module> main(args) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 158, in main trainer.fit(model, data_module, ckpt_path=ckpt_path) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 544, in fit call._call_and_handle_interrupt( File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\call.py", line 68, in _call_and_handle_interrupt trainer._teardown() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1012, in _teardown self.strategy.teardown() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\ddp.py", line 406, in teardown super().teardown() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\parallel.py", line 127, in teardown super().teardown() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 528, in teardown self.lightning_module.cpu() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 79, in cpu return super().cpu() File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 954, in cpu return self._apply(lambda t: t.cpu()) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply module._apply(fn) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply module._apply(fn) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply param_applied = fn(param) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\runtime\lib\site-packages\torch\nn\modules\module.py", line 954, in <lambda> return self._apply(lambda t: t.cpu()) RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Epoch 0: 0%| | 0/15226 [00:00<?, ?it/s] C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>After set cpu as train device, I could see the main error:
try: # 使用正则表达式匹配文件名中的数字部分，并按数字大小进行排序 newest_ckpt_name = get_newest_ckpt(os.listdir(ckpt_dir)) ckpt_path = ckpt_dir / newest_ckpt_name except Exception: ckpt_path = None print("ckpt_path:", ckpt_path) trainer.fit(model, data_module, ckpt_path=ckpt_path) C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2>set CUDA_LAUNCH_BLOCKING=1 && runtime\python.exe GPT_SoVITS/s1_train.py --config_file "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\TEMP/tmp_s1.yaml" Seed set to 1234 Using 16bit Automatic Mixed Precision (AMP) GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs <All keys matched successfully> Traceback (most recent call last): File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 186, in <module> main(args) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\s1_train.py", line 159, in main newest_ckpt_name = get_newest_ckpt(os.listdir(ckpt_dir)) File "C:\Users\Naozumi0512\Desktop\AI\GPT-SoVITS-beta0217fix2\GPT_SoVITS\AR\utils\__init__.py", line 24, in get_newest_ckpt newest_ckpt = sorted_info[0][2] IndexError: list index out of rangeWhat I should do? Referring to the wiki I thought I could jump right in after adding the new language g2p, but I got much more than that.
sovits训练：text_emb shape对不上，可以删了这个key，或者把缺的shape用0填充 gpt训练：text_emb根据yaml config里会预留到512个symbol的位置，如果超过了这个emb会索引越界，试试把这个参数加大点，如果load dict出问题同样试试微操text_emb key
https://huggingface.co/AkitoP/GPT-SoVITS-JA-ProsodyControl_model/blob/main/insert_symbol.ipynb 这里有微操text emb key的代码。GPT训练环节需要yaml config里面加phoneme_vocab_size，如果512个音素不够用的话。
https://github.com/RVC-Boss/GPT-SoVITS/wiki/%E8%AE%AD%E7%BB%83%E6%96%B0%E8%AF%AD%E8%A8%80(how-to-train-the-models-with-other-languages)
So I'm trying to implement the Spanish language by fine-tuning the base model. I've created a G2P system and updated the language symbol set. I'm using Espeak IPA phonemizer for it. Now my question is, should the IPA output be joined together or separated as individual characters as shown in the image posted below?
