Syno8
Syno8
``> 修改`layer_range`参数，样例的脚本是微调了第0层和第14层，多传一些数字就是微调更多层，不传这个参数默认就是微调所有层。增大`lora_rank`也可以增加微调参数。 能直接微调整个模型吗？我尝试了去掉ues_lora 出了点问题 ```IndexError: list index out of range self.optimizer = self._configure_zero_optimizer(basic_optimizer) optimizer = DeepSpeedZeroOptimizer( File "/path/to/anaconda3/envs/py10/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 270, in __init__ File "/path/to/anaconda3/envs/py10/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1444, in _configure_zero_optimizer self.dtype =...
@minlo-18 这个损失最后能训练到多少停止呢？直接训练300步就确定收敛了吗？我训练了一个模型，用生成来做2分类任务，最后训练得到的损失是total loss 2.990189E-02 | loss 2.990189E-02 | loss scale 32768.0 |speed 454.12 samples/(min*GPU) 感觉这个数值太小了，有些不太正常
I mkdir first as follows and save the model correctly. `os.makedirs(os.path.dirname(args.save_path), exist_ok=True)`
Do you have solved this problem?
@recool08 你有找到问题吗？ @kurisusnowdeng do you have report your results on chatgpt training
这个是原模型还是你微调后的模型？
chatgpt can generate the format text and image. this need to keep the pertaining data in original format
@timongurcke I found that batch decoding leads to a error, I think it is caused by 'pad' `outputs = model.generate(**batch, max_new_tokens=training_args.gen_length, do_sample=False, num_return_sequences=1)`