Patsakula Nikita comments

Results 6 comments of


Patsakula Nikita

WIP: Build script fixed: includes separated from symbols.

@LaurentMazare, hello! Can i ask for review?

WIP: Build script fixed: includes separated from symbols.

> Looks reasonable, could you rebase this branch? Hopefully this should get rid of the clippy error in the CI. Hello @LaurentMazare! I rebased my branch on new `main`, added...

WIP: NLLB support.

Hello @guillaume-be! I've made a few assumptions that need a sanity check: 1. According to [this](https://huggingface.co/facebook/nllb-200-distilled-600M/blob/main/config.json#L5) line I assumed that encoder and decoder of M2M should be NLLB compatible. 2....

Use AsRef<Path> instead of &str.

> Thank you @npatsakula for this PR. > > I would be in favor of actually moving from `&str` to `Path` instead of `AsRef` for 2 reasons: > > 1....

Use AsRef<Path> instead of &str.

Hello @guillaume-be! I reverted to `AsRef` (doc-comments too).

DRAFT: NLLB tokenizer support.

Hello @guillaume-be! I bring some news about adding NLLB support: 1. I added support of special tokens configuration in https://github.com/guillaume-be/rust-tokenizers/pull/76/commits/c13ff10799389e0a6d98d887f36a3cf75880eeb8: it allows to work with config files such [this](https://huggingface.co/facebook/nllb-200-distilled-600M/blob/main/special_tokens_map.json). I...