windows 系统训练失败，数据集字符编码问题

[yjc980121]

Describe the bug

[WARNING:modelscope] Reusing dataset dataset_builder (C:\Users\DELL.cache\modelscope\hub\datasets\iic\ms_bench\master\data_files) [INFO:modelscope] Generating dataset dataset_builder (C:\Users\DELL.cache\modelscope\hub\datasets\iic\ms_bench\master\data_files) [INFO:modelscope] Reusing cached meta-data file: C:\Users\DELL.cache\modelscope\hub\datasets\iic\ms_bench\master\data_files\2b408f043079b23300a89e65c7a2d027 Traceback (most recent call last): File "C:\Python311\Lib\site-packages\swift\cli\sft.py", line 5, in sft_main() File "C:\Python311\Lib\site-packages\swift\utils\run_utils.py", line 31, in x_main result = llm_x(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\swift\llm\sft.py", line 113, in llm_sft train_dataset, val_dataset = get_dataset( ^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\swift\llm\utils\dataset.py", line 1349, in get_dataset dataset = get_function( ^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\swift\llm\utils\dataset.py", line 267, in get_dataset_from_repo dataset = load_ms_dataset(dataset_id, subset_split_list) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\swift\llm\utils\dataset.py", line 213, in load_ms_dataset dataset = MsDataset.load( ^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\swift\llm\utils\utils.py", line 100, in _msdataset_ddp_load dataset = _old_msdataset_load(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\ms_dataset.py", line 315, in load dataset_inst = remote_dataloader_manager.load_dataset( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\data_loader\data_loader_manager.py", line 132, in load_dataset oss_downloader.process() File "C:\Python311\Lib\site-packages\modelscope\msdatasets\data_loader\data_loader.py", line 83, in process self._prepare_and_download() File "C:\Python311\Lib\site-packages\modelscope\msdatasets\data_loader\data_loader.py", line 147, in _prepare_and_download self.dataset = self.data_files_manager.fetch_data_files( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\data_files\data_files_manager.py", line 116, in fetch_data_files return builder.as_dataset() ^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\download\dataset_builder.py", line 249, in as_dataset return DatasetDict({ ^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\download\dataset_builder.py", line 250, in k: self._convert_csv_to_dataset(k, v) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\modelscope\msdatasets\download\dataset_builder.py", line 218, in _convert_csv_to_dataset df = pd.read_csv( ^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\datasets\streaming.py", line 75, in wrapper return function(*args, download_config=download_config, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\datasets\download\streaming_download_manager.py", line 784, in xpandas_read_csv return pd.read_csv(xopen(filepath_or_buffer, "rb", download_config=download_config), **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 1024, in read_csv return _read(filepath_or_buffer, kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 618, in _read parser = TextFileReader(filepath_or_buffer, **kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 1618, in init self._engine = self._make_engine(f, self.engine) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 1896, in _make_engine return mapping[engine](f, **self.options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 93, in init self._reader = parsers.TextReader(src, **kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "parsers.pyx", line 574, in pandas._libs.parsers.TextReader.cinit File "parsers.pyx", line 663, in pandas._libs.parsers.TextReader._get_header File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status File "parsers.pyx", line 2053, in pandas._libs.parsers.raise_parser_error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 128: invalid start byte

Your hardware and system info Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等) torch==2.0.1+cu118 windows 11 RTX 3060

看描述，是因为读取过程中windows 默认的gbk编码，加载的时候应该是按照utf8 识别 看看能不能兼容下windows系统和linux系统质检编码的兼容性，从报错信息来看，应该是datasets这个库导致的，但是这个库是huggingface的库，不知道有没有办法解决

[Rundong-Li]

同问

[catundchat]

win10上也有类似问题。在使用已有的数据集例如ms-bench, 非自定义数据训练时存在报错：UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 128: invalid start byte

[Rundong-Li]

win10上也有类似问题。在使用已有的数据集例如ms-bench, 非自定义数据训练时存在报错：UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 128: invalid start byte

我的系统就是win10，已经切换到Linux了（WSL2），没有出现这个问题