RRHF icon indicating copy to clipboard operation
RRHF copied to clipboard

resize embedding after add_special_tokens

Open Switchsyj opened this issue 5 months ago • 0 comments

Hi, thanks for your great work! I would like to point out a potential bug in this code: add_special_tokens without checking embedding size is very dangerous especially for llama. In fact, llama use <end_of_text> as eos and bos token during training. Otherwise, you need to resize the embedding after add_special_tokens '' or it would out of bounds while torch.gather.

Code line: https://github.com/GanjinZero/RRHF/blob/e1a2b61f7d91fbee4cfaa3923327fcc5c5c733de/train.py#L302

Switchsyj avatar Sep 03 '24 03:09 Switchsyj