Oscar Sainz Jimenez

icon indicating a website link osainz59.github.io icon indicating an email [email protected]

icon company University of the Basque Country (UPV/EHU) icon location PhD student on NLP.

Results 1 comments of Oscar Sainz Jimenez

LLaMA FastTokenizer does not add `eos_token_id` at the end.

> But it shouldn't add an `eos` token right? The LM is not trained to generate a token after the `eos` I believe. By default, but if specified with `add_eos_token=True`...