sensevoice微调方言

[AlisonDexter]

你好，请问你训练一个新语言识别模型时，这个train_text_language.txt中的标签是如何给值的，是给zh吗？还是可以自定义一个新的呢？

[EdwardGgZ]

你好，请问你训练时运行bash finetune.sh时，++model等参数能传递到train_ds.py中吗，我一直报错

[AlisonDexter]

改用whisper微调方言了

EdwardGgZ @.***> 于2025年6月12日周四 19:58写道：

EdwardGgZ left a comment (FunAudioLLM/SenseVoice#225) https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-2966406160

你好，请问你训练时运行bash finetune.sh时，++model等参数能传递到train_ds.py中吗，我一直报错

image.png (view on web) https://github.com/user-attachments/assets/76b677ce-8dcb-41c3-ae78-1b7329a9f29a

— Reply to this email directly, view it on GitHub https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-2966406160, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCDNNFGKPJD6JWDXELDOQ3D3DFTP7AVCNFSM6AAAAAB55ZCWZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRWGQYDMMJWGA . You are receiving this because you authored the thread.Message ID: @.***>

[liuxu999]

@AlisonDexter 嗨嗨你好，请问你当时用sensevoice微调新方言语种的时候，怎么设置新方言语种的类型呢，用zh嘛。我目前正在微调闽南语

[AlisonDexter]

用zh

liuxu999 @.***> 于2025年6月30日周一 16:28写道：

liuxu999 left a comment (FunAudioLLM/SenseVoice#225) https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-3018258013

@AlisonDexter https://github.com/AlisonDexter 嗨嗨你好，请问你当时用sensevoice微调新方言语种的时候，怎么设置新方言语种的类型呢，用zh嘛。我目前正在微调闽南语

— Reply to this email directly, view it on GitHub https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-3018258013, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCDNNFGR65TUR2OJDQTEU5T3GDYKNAVCNFSM6AAAAAB55ZCWZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMJYGI2TQMBRGM . You are receiving this because you were mentioned.Message ID: @.***>

[liuxu999]

@AlisonDexter 我对sensevoice_small添加了minnan语的token，然后用大概2万条闽南语言训练后，发现指标相较于微调前才增加了4%。请问你微调方言用sensevoice_small和whisper这两个模型的效果如何呢

[AlisonDexter]

我觉得whisper比较好，但是力大飞砖。出现微调四川话后对原有普通话的识别效果下降的情况， whisper支持普通话和粤语，我了解到的有的人会在训练方言会根据片区来训练。

liuxu999 @.***> 于2025年7月3日周四 14:55写道：

liuxu999 left a comment (FunAudioLLM/SenseVoice#225) https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-3031095874

@AlisonDexter https://github.com/AlisonDexter 我对sensevoice_small添加了minnan语的token，然后用大概2万条闽南语言训练后，发现指标相较于微调前才增加了4%。请问你微调方言用sensevoice_small和whisper这两个模型的效果如何呢

— Reply to this email directly, view it on GitHub https://github.com/FunAudioLLM/SenseVoice/issues/225#issuecomment-3031095874, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCDNNFHQUBN3LXUK2CSWSWD3GTHV3AVCNFSM6AAAAAB55ZCWZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMZRGA4TKOBXGQ . You are receiving this because you were mentioned.Message ID: @.***>

[lukeewin]

@AlisonDexter 我对sensevoice_small添加了minnan语的token，然后用大概2万条闽南语言训练后，发现指标相较于微调前才增加了4%。请问你微调方言用sensevoice_small和whisper这两个模型的效果如何呢

请问你在微调闽南语后，会影响原来对普通话的识别能力吗？你是设置语种标签为zh还是minnan？请问你是如何添加minnan语的token

[wangchao112211]

我也再搞方言，但是我会报一个说不支持接着训练的错误，也不知道怎么回事： [2025-07-15 09:27:15,440][root][INFO] - total_num of samplers: 226156, /data/ASR/SenseVoice/data/train_example.jsonl [2025-07-15 09:27:15,440][root][INFO] - total_num of samplers: 6, /data/ASR/SenseVoice/data/val_example.jsonl [2025-07-15 09:27:15,466][root][INFO] - total_num of samplers: 226156, /data/ASR/SenseVoice/data/train_example.jsonl [2025-07-15 09:27:15,466][root][INFO] - total_num of samplers: 6, /data/ASR/SenseVoice/data/val_example.jsonl Error executing job with overrides: ['++model=iic/SenseVoiceSmall', '++trust_remote_code=true', '++train_data_set_list=/data/ASR/SenseVoice/data/train_example.jsonl', '++valid_data_set_list=/data/ASR/SenseVoice/data/val_example.jsonl', '++dataset_conf.data_split_num=1', '++dataset_conf.batch_sampler=BatchSampler', '++dataset_conf.batch_size=2000', '++dataset_conf.sort_size=1024', '++dataset_conf.batch_type=token', '++dataset_conf.num_workers=0', '++train_conf.max_epoch=50', '++train_conf.log_interval=1', '++train_conf.resume=true', '++train_conf.validate_interval=2000', '++train_conf.save_checkpoint_interval=2000', '++train_conf.keep_nbest_models=20', '++train_conf.avg_nbest_model=10', '++train_conf.use_deepspeed=false', '++train_conf.deepspeed_config=/data/ASR/SenseVoice/deepspeed_conf/ds_stage1.json', '++optim_conf.lr=0.0002', '++output_dir=./outputs'] [rank1]: Traceback (most recent call last): [rank1]: File "/data/FunASR/funasr/bin/train_ds.py", line 244, in [rank1]: main_hydra() [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main [rank1]: _run_hydra( [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra [rank1]: _run_app( [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app [rank1]: run_and_report( [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report [rank1]: raise ex [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report [rank1]: return func() [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in [rank1]: lambda: hydra.run( [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run [rank1]: _ = ret.return_value [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value [rank1]: raise self._return_value [rank1]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job [rank1]: ret.return_value = task_function(task_cfg) [rank1]: File "/data/FunASR/funasr/bin/train_ds.py", line 56, in main_hydra [rank1]: main(**kwargs) [rank1]: File "/data/FunASR/funasr/bin/train_ds.py", line 155, in main [rank1]: trainer.resume_checkpoint( [rank1]: File "/data/FunASR/funasr/train_utils/trainer_ds.py", line 471, in resume_checkpoint [rank1]: self.start_epoch = checkpoint["epoch"] [rank1]: KeyError: 'epoch' Error executing job with overrides: ['++model=iic/SenseVoiceSmall', '++trust_remote_code=true', '++train_data_set_list=/data/ASR/SenseVoice/data/train_example.jsonl', '++valid_data_set_list=/data/ASR/SenseVoice/data/val_example.jsonl', '++dataset_conf.data_split_num=1', '++dataset_conf.batch_sampler=BatchSampler', '++dataset_conf.batch_size=2000', '++dataset_conf.sort_size=1024', '++dataset_conf.batch_type=token', '++dataset_conf.num_workers=0', '++train_conf.max_epoch=50', '++train_conf.log_interval=1', '++train_conf.resume=true', '++train_conf.validate_interval=2000', '++train_conf.save_checkpoint_interval=2000', '++train_conf.keep_nbest_models=20', '++train_conf.avg_nbest_model=10', '++train_conf.use_deepspeed=false', '++train_conf.deepspeed_config=/data/ASR/SenseVoice/deepspeed_conf/ds_stage1.json', '++optim_conf.lr=0.0002', '++output_dir=./outputs'] [rank0]: Traceback (most recent call last): [rank0]: File "/data/FunASR/funasr/bin/train_ds.py", line 244, in [rank0]: main_hydra() [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main [rank0]: _run_hydra( [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra [rank0]: _run_app( [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app [rank0]: run_and_report( [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report [rank0]: raise ex [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report [rank0]: return func() [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in [rank0]: lambda: hydra.run( [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run [rank0]: _ = ret.return_value [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value [rank0]: raise self._return_value [rank0]: File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job [rank0]: ret.return_value = task_function(task_cfg) [rank0]: File "/data/FunASR/funasr/bin/train_ds.py", line 56, in main_hydra [rank0]: main(**kwargs) [rank0]: File "/data/FunASR/funasr/bin/train_ds.py", line 155, in main [rank0]: trainer.resume_checkpoint( [rank0]: File "/data/FunASR/funasr/train_utils/trainer_ds.py", line 471, in resume_checkpoint [rank0]: self.start_epoch = checkpoint["epoch"] [rank0]: KeyError: 'epoch' E0715 09:27:18.961000 140497700328000 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 106632) of binary: /data/anaconda/envs/cosyvoice/bin/python3.10 Traceback (most recent call last): File "/data/anaconda/envs/cosyvoice/bin/torchrun", line 8, in sys.exit(main()) File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 347, in wrapper return f(*args, **kwargs) File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main run(args) File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run elastic_launch( File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/data/anaconda/envs/cosyvoice/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/data/FunASR/funasr/bin/train_ds.py FAILED

Failures: [1]: time : 2025-07-15_09:27:18 host : localhost.localdomain rank : 1 (local_rank: 1) exitcode : 1 (pid: 106633) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure): [0]: time : 2025-07-15_09:27:18 host : localhost.localdomain rank : 0 (local_rank: 0) exitcode : 1 (pid: 106632) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

(cosyvoice) [root@localhost SenseVoice]#

[wwfcnu]

可以贴一下你们微调时funasr和modelscope的版本吗，我这边一直报错 Traceback (most recent call last): File "/mnt/lustre/wangweifei/repository/asr/SenseVoice/demo1.py", line 12, in model = AutoModel( File "/mnt/lustre/wangweifei/repository/asr/FunASR/funasr/auto/auto_model.py", line 125, in init model, kwargs = self.build_model(**kwargs) File "/mnt/lustre/wangweifei/repository/asr/FunASR/funasr/auto/auto_model.py", line 228, in build_model tokenizer = tokenizer_class(**tokenizer_conf) TypeError: 'NoneType' object is not callable

[wwfcnu]

想问一下微调时，训练集和验证集的比例设置多少合适呀