Arthur

I converted the vocab to the appropriate format to read it, sorry I forgot that step. Will push it now. I used your tokenizer's `encoder`

I used this : ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-5-world-1b5", trust_remote_code=True) with open("/Users/arthurzucker/Work/transformers/rwkv.txt", "wb") as f: for index, token in tokenizer.encoder.item(): f.write(token + b"\n") tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt") prompt...

Pushed the tokenizer here: https://huggingface.co/ArthurZ/rwkv-5

Cool model! But as long as we don't have official/ very good pretraining checkpoints, not really anything we can do!

My recommendation would be to put the model on the hub following [this tutorial](https://huggingface.co/docs/transformers/custom_models), which will help having a working code without going trough the hassle of all the review...

I can't really reproduce this and have not seen this anywhere else. The OS Error suggests that the interface is not available, meaning that most probably the path to your...

Hey! Thanks for reporting could you share the entire traceback of the error? 😉

Hey! I did not have time to check this, if you can isolate a small reproduction script (without all the training loop) would be great. Otherwise, I am investigating

Ok, low on bandwidth so pinging @Rocketknight1 in case he can have a look!

Sorry @hexie1995 did not have time to have look 😢