Nicolas Patry

For the tokenizer item I suggest using https://github.com/huggingface/tokenizers/ Should work out of the box once converted (when this PR lands: https://github.com/huggingface/transformers/pull/21955 it should become a simple `let tokenizer = Tokenizer::from_file("filename")`...

Good news everyone ! https://github.com/huggingface/tokenizers/pull/1183 (If this goes, I'll try to make a release soon after)

@dnlmlr Do you have bench to back that up ? I didn't found that to be the case whenever I tried. Memory-mapping was always consistently better than reading a file...

Closing as stale. Thanks for the contribution Nick, happy to take some back now that we're back on Apache ! Cheers.

Not in this PR, this PR is the dirty work, there's a lot of legwork but yes all models will be supported as much out of the box as possible

Hi, Disclaimer: CPU support is on 'best-effort' only. The reason is that it's a very different problem space with what we're trying to solve here, things like offloading and such...

The first screenshot you shared says that you're trying to write on a read-only volume. Meaning it's most likely a permission problem onto your ` data` folder. The second screenshot...

Is `all` expected at all ? https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars Otherwise, there's no need to do parsing of anything, returning `None` here should take care of using all available GPUs ! However, unless...

For the doc you linked the env variable is `NVIDIA_VISIBLE_DEVICES` not `CUDA_VISIBLE_DEVICES`. Maybe that explains it ? Maybe we should even just not look at any nvidia env variable and...

> The concern I would raise here that it may cause implicit behavior changes for existing setups - how do you feel about this? It's already the case with `CUDA_VISIBLE_DEVICES`....