DeepSpeedExamples icon indicating copy to clipboard operation
DeepSpeedExamples copied to clipboard

fix: don't add eot token if add_eot_token knob is False

Open EeyoreLee opened this issue 1 year ago • 0 comments

  1. add a arg named add_eot_token to avoid to add eot token when args.add_eot_token is False. (fix step2 llama2)
  2. remove some redundant code. like chosen_token["input_ids"] = chosen_token["input_ids"]
  3. avoid using list as python function args default value.

EeyoreLee avatar Dec 20 '23 07:12 EeyoreLee