Remove the fixed `eot_token` mechanism for SFT

[Xingfu-Yi]

Background

Not all pretrained LLMs use <|endoftext|> as the eot_token, therefore it's inappropriate to fix it.

Changes

• Removed the hardcoded eot_token: args.end_of_conversation_token = "<|endoftext|>".
• Added a new argument in the parser called eot_token which is <|endoftext|> by default. Users can manually set the token according to the pretrained model they use.

[Xingfu-Yi]

Hi @arashb, @duli2012, @awan-10, @eltonzheng,

I hope you're doing well. When you have a moment, could you kindly take a look at this PR? It has already received one approval, but it seems to be stuck and needs further reviews to move forward.

Thank you so much in advance for your time and help.

Best regards,
Yi

[loadams]

Hi @arashb, @duli2012, @awan-10, @eltonzheng,

I hope you're doing well. When you have a moment, could you kindly take a look at this PR? It has already received one approval, but it seems to be stuck and needs further reviews to move forward.

Thank you so much in advance for your time and help.

Best regards, Yi

Hi @Xingfu-Yi - we will work on getting this PR merged, sorry for the delay.