Vincent Nguyen

do you confirm that the only multilingual datasets are the same as llama ie. all the 20 language wikipedia stuff ? thanks

Well the appendix is not so clear. It says for mc4 it was filtered on English. But for Redpajamas it does not say and it includes Wikipedia. The goal of...

Ok then it will not be as good as llama for multilingual ability (conversation in another language) or translation. It's too bad and remove a non negligible part of the...

I will, but even when finetuning with wikipedia + cc-net with 3 languages (EN/DE/FR) the loss remains a little high. I will test further with translation finetuning.

It is not implemented yet. I recently revamped beam_search.py and you can check where we use n_best to make sure we add enough hypotheses. You can easily do the same...

Thanks. Have you tried to run twice the same run? not getting the same results in multi gpus ?

I am pretty sure you are asking the same question as what I am looking at (if not sorry to hijack this post). I read the paper Model soup mentioned...

@francoishernandez @pltrdy do you recall why this https://github.com/OpenNMT/OpenNMT-py/pull/1108 was introduced ? threads memoery leakages ?

actually behavior seems normal according to this: https://github.com/google/sentencepiece/blob/master/doc/special_symbols.md it says: " Control symbols must be inserted outside of the SentencePiece segmentation." SO then, I don't understand the insertion of sep2...

closing now it's merged with #2198