DeepSpeedExamples
DeepSpeedExamples copied to clipboard
fix: don't add eot token if add_eot_token knob is False
- add a arg named
add_eot_tokento avoid to add eot token whenargs.add_eot_tokenis False. (fix step2 llama2) - remove some redundant code. like
chosen_token["input_ids"] = chosen_token["input_ids"] - avoid using list as python function args default value.