自定义数据集报错

[Vindicator645]

文档未明确说明custom_train_dataset_path废弃后应如何指定自定义数据集，我使用--custom_train_dataset_path 或者提示的--dataset，均会报错： File "/home/work_nfs7/t/swift519/swift-main/swift/llm/utils/dataset.py", line 1361, in get_local_dataset return _post_preprocess(dataset, dataset_sample, random_state, None, dataset_test_ratio, remove_useless_columns) File "/home/work_nfs7/t/swift519/swift-main/swift/llm/utils/dataset.py", line 311, in _post_preprocess train_dataset, val_dataset = train_dataset.train_test_split( File "/home/environment/t/anaconda3/envs/swift520/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 558, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) File "/home/environment/t/anaconda3/envs/swift520/lib/python3.9/site-packages/datasets/fingerprint.py", line 482, in wrapper out = func(dataset, *args, **kwargs) File "/home/environment/t/anaconda3/envs/swift520/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 4412, in train_test_split raise ValueError( ValueError: test_size=1 should be either positive and smaller than the number of samples 1 or a float in the (0, 1) range

Experimental environment: V100, A10, 3090

21GB GPU memory

PYTHONPATH=../../..
CUDA_VISIBLE_DEVICES=4,5,6,7
python llm_sft.py
--model_id_or_path qwen/Qwen-Audio-Chat
--model_revision master
--sft_type lora
--tuner_backend peft
--template_type AUTO
--dtype AUTO
--output_dir output
--dataset /home/work_nfs7/swift-main/asset/dataset/train.jsonl
--dataset_test_ratio 0.5
--train_dataset_sample 1000
--num_train_epochs 100
--max_length 2048
--check_dataset_strategy warning
--lora_rank 8
--lora_alpha 32
--lora_dropout_p 0.05
--lora_target_modules DEFAULT
--gradient_checkpointing true
--batch_size 10
--weight_decay 0.1
--learning_rate 1e-4
--gradient_accumulation_steps 16
--max_grad_norm 0.5
--warmup_ratio 0.03
--eval_steps 100
--save_steps 100
--save_total_limit 2
--logging_steps 10
--use_flash_attn false
--lazy_tokenize true \