CrispStrobe
CrispStrobe
thanks for having a look at this! indeed, since level=DEBUG is off per default now, we no longer see the "system info:" where there was a difference visible in both...
thanks. probably i described the problem in too unprecise words, sorry for that. is is dependend upon how ollama starts and will keep this way forever, not be faster some...
thanks, version is 546.12, and i had no bsods yet with eg lm studio
thank you. it seems to work well at the moment (and after updating to version 0.1.31).
sorry should have been in the screenshot, think this was NousHermes-2-Mistral-7B-DPO (4bit)
in the meanwhile, there is also a fix for the pretokenizer. i have included it in [this](https://github.com/CrispStrobe/llm_scripts/blob/main/quantize-with-pretokenizer-fix.ipynb) [Kaggle](https://www.kaggle.com/code/chr1str/quantize-with-pretokenizer-fix) notebook. of course you can adapt it if you wish.
indeed might be better to wait with regard to the pre-tokenizer. i am not completely sure i understood the procedure for new models like say llama3 merges. but my current...
many thanks, perfect - and wow, that was swift!
would it be possible to adapt an existing eg 7b mistral v0.1/0.2 based finetuned model so that it works with the new extended 32768 vocabulary and the v3 tokenizer, and...
The documentation already shows how to do it in python, but maybe you mean per cli? I just tried it with: `pqa --llm "ollama/cas/llama3.1-8b-spaetzle-v74" --llm_config '{"model_list": [{"model_name": "ollama/cas/llama3.1-8b-spaetzle-v74", "litellm_params": {"model":...