Lockinwize Lolite

Get another bug when converting the image encoder to gguf. Log: ```sh python3 ./examples/minicpmv/convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2 --llava-projector ../MiniCPM-V-2/llava.projector --output-dir ../MiniCPM-V-2-GGUF --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5 gguf: This...

BTW it may be better if you can add `device_map="auto"` in `minicpm-surgery.py`#L12&L42 :) It can take full advantages of a GPU :)

> I followed `examples/llava/convert-image-encoder-to-gguf.py`. It seems that they also don't use `.cpu()` here, and in my environment the model are loaded to CPU by default. I fixed my problems by...

Get another bug. Log: ```sh > python3 ./examples/minicpmv/minicpm-surgery.py -m ../MiniCPM-V-2 Loading checkpoint shards: 100% 2/2 [00:34 python3 ./examples/minicpmv/convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2 --llava-projector ../MiniCPM-V-2/llava.projector --output-dir ../MiniCPM-V-2-GGUF --image-mean 0.5 0.5 0.5 --image-std 0.5...

> Does your MiniCPM-V-2 folder have `tokenizer.json`? It is a newly uploaded file in https://huggingface.co/openbmb/MiniCPM-V-2/tree/main. Yes, I confirm that. Log: ```sh > ls ../MiniCPM-V-2/ -alh total 8.0G drwxr-xr-x 4 root...

> So it seems that the save_pretrained method in surgery.py do not save the tokenizer.json file. I manually copy the tokenizer.json file into the MiniCPM sub-folder before the tokenizer.json is...

Convert successfully, thx! However, I got a bad test result... Log here: ```sh > ./minicpmv-cli -ngl 1000000 -m ./MiniCPM-V-2-GGUF/MiniCPM-V-2.F16.gguf --mmproj ./MiniCPM-V-2-GGUF/mmproj-model-f16.gguf -c 4096 --temp 0.6 --top-p 0.8 --top-k 100 --repeat-penalty...

@Achazwl Can you help test the model I quantized? Link here:

> The link you provided only contains fp16 models The mmproj gguf model is actually there, I just rename it :) Link to the mmproj gguf model:

I did some further tests. When I use only the CPU, the model's output is very, very normal. However, when I switching to the GPU, the model seemed... mad. Tested...