ChatLM-mini-Chinese icon indicating copy to clipboard operation
ChatLM-mini-Chinese copied to clipboard

运行3.4python ptr_train.py时报错OSError: Can't load tokenizer for 'D:/pycharmenv/ChatLM-mini-Chinese/model_save/'.

Open summerFF opened this issue 6 months ago • 2 comments

运行3.4python ptr_train.py时报错 (base) D:\pycharmenv\ChatLM-mini-Chinese>python pre_train.py D:\SOFTWAREE\anacondaa\Lib\site-packages\transformers\utils\generic.py:441: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( D:\SOFTWAREE\anacondaa\Lib\site-packages\transformers\utils\generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( Traceback (most recent call last): File "D:\pycharmenv\ChatLM-mini-Chinese\pre_train.py", line 136, in pre_train(config) File "D:\pycharmenv\ChatLM-mini-Chinese\pre_train.py", line 58, in pre_train tokenizer = PreTrainedTokenizerFast.from_pretrained(config.tokenizer_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\SOFTWAREE\anacondaa\Lib\site-packages\transformers\tokenization_utils_base.py", line 2012, in from_pretrained raise EnvironmentError( OSError: Can't load tokenizer for 'D:/pycharmenv/ChatLM-mini-Chinese/model_save/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'D:/pycharmenv/ChatLM-mini-Chinese/model_save/' is the correct path to a directory containing all relevant files for a PreTrainedTokenizerFast tokenizer.

summerFF avatar Jul 27 '24 14:07 summerFF