lenicolas

Courtesy of @lynxpda : OOV translation, relative position encoding and other activation functions

3

1. argument "byte-fallback-off" disables out-of-vocabulary as-is tranlsation. Default is enabled. 2. absolute position encoding is default, but one can enable relative position encoding (Shaw type) using the config.json file (put...

Advanced filters, transforms, reliable deduplication, versatile SBD, updated NLLB list, eval with pivot and translation memoirs, more custom transformers, byte-fallback

1

1. New "utils" dir includes flores200_dataset dir 2. Update and edits to NLLB languages list 3. Paste translation memoirs in flores200_dataset dir (in a {dataset} subdir as {nllb_langs[lang_code]}.{dataset} files) 4....

Hello, I have been producing a French-Chinese model a few months ago, and noticed that byte-fallback was yielding strange triplets of BF tokens... then, lately, to debug both Chinese and...