I tried to pre-tokeinzation but failed

[userdsr]

I tried to run 'python -m ochat.data.generate_dataset --model-type MODEL_TYPE --model-path BASE_REPO --in-files data.jsonl --out-prefix PRETOKENIZED_DATA_OUTPUT_PATH' but failed:

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/public/home/szlab_daisr/Deploy/openchat/ochat/data/generate_dataset.py", line 167, in generate_dataset(**vars(args)) File "/public/home/szlab_daisr/Deploy/openchat/ochat/data/generate_dataset.py", line 149, in generate_dataset generate_split(model_type, model_path, train_conversations, "train", out_prefix, per_sequence_loss) File "/public/home/szlab_daisr/Deploy/openchat/ochat/data/generate_dataset.py", line 131, in generate_split parquet.write_table(pyarrow.concat_tables([ray.get(handle) for handle in handles]), f"{out_prefix}.{split_name}.parquet") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/public/home/szlab_daisr/Deploy/openchat/ochat/data/generate_dataset.py", line 131, in parquet.write_table(pyarrow.concat_tables([ray.get(handle) for handle in handles]), f"{out_prefix}.{split_name}.parquet") ^^^^^^^^^^^^^^^ File "/public/home/szlab_daisr/anaconda3/envs/openchat2/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/public/home/szlab_daisr/anaconda3/envs/openchat2/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/public/home/szlab_daisr/anaconda3/envs/openchat2/lib/python3.11/site-packages/ray/_private/worker.py", line 2624, in get raise value.as_instanceof_cause() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/public/home/szlab_daisr/anaconda3/envs/openchat2/lib/python3.11/site-packages/ray/exceptions.py", line 148, in as_instanceof_cause class cls(RayTaskError, cause_cls): TypeError: type 'pydantic_core._pydantic_core.ValidationError' is not an acceptable base type