gpt_tokenize: uknown token

[ShJavokhir]

I just installed everything perfectly without any problems. But when I try to run I am facing this issue.

[ShJavokhir]

It looks like got stuck in a something like loop. Because it was constantly eating CPU and RAM resourses

[hu0573]

me too

[hu0573]

I think windows is not a good choice, I run it on ubuntu now, everything smooth!

[mabry1985]

seeing this on an m1 mac right now

LLAMA_EMBEDDINGS_MODEL=./models/ggml-model-q4_1.bin
MODEL_PATH=./models/ggml-gpt4all-l13b-snoozy.bin

[mabry1985]

This fixed it for me https://github.com/imartinez/privateGPT/issues/13#issuecomment-1546769749

[NPap0]

Getting the same error, gpt_tokenize: unknown token 'Γ' gpt_tokenize: unknown token 'Ç' gpt_tokenize: unknown token 'Ö' gpt_tokenize: unknown token 'Γ' gpt_tokenize: unknown token 'Ç' gpt_tokenize: unknown token 'Ö' gpt_tokenize: unknown token 'Γ' gpt_tokenize: unknown token 'Ç' gpt_tokenize: unknown token 'Ö' CPU almost at 100% and memory usage slowly rising so it must still be working but I get no output. Running on Windows. I have the same model type running and have correctly named it in the .env file (GPT4ALL) but I'll be switching to Llama.cpp to check

EDIT: I did get an output after 5ish minutes? So something is going on and its making it terribly slow.

[thekit]

I also get an output eventually after seeing the warnings.

[NULLOVE]

I have the same question when I run GPT4All on windows,then ,how to solve this problem?