Nicolas Patry
Nicolas Patry
Hi Thanks for this PR. Do you mind splitting the feature gates from the dependencies update in separate PRs ? It's much easier for me to add the cli-feature gate...
Thanks you for changing the PR !
It seems like a sound idea, since the merge table is fused in sorted order, we should in theory be able to append only from partial results (read -> rencoded...
Hi @cccntu , The tokenizer never knows anything about the embeddings. It has no clue whatsoever. AFAIK, fine-tuning a model, you never touch the tokenizer itself. If you do, and...
Unfortunately, no, this library is wrote in rust, exclusively for performance. There's also no clean way I can point you toward the code either since this would touch quite many...
Hi @agni-ai Can you try with the latest `0.11.1` ? It should work on old libc again.
Hi @Eleo22 I have no idea what ` from transformers_domain_adaptation import VocabAugmentor` Could you maybe take that up to `transformers_domain_adaptation` repository ? Or figure out a reproducible example without this...
I beg to differ, if you want to install from source you probably are a bit knowledgeable. How sources and git works is beyond the simple commands which focus on...
Okay, I might be too heavily biased. Would like to do a PR ? The file that should be changed is here: https://github.com/huggingface/tokenizers/blob/master/docs/source/installation/python.inc I think limiting this a simple comment...
Seems perfectly fine to me.