Error: "self and mat2 must have the same dtype, but got Half and Char" when running in 8bit quantization

[rudolphos]

This is the error I get:

self and mat2 must have the same dtype, but got Half and Char

when I pass the image path to my script in 8-bit quantization mode, which according to the official website is possible. I have no issues when I run the full FP16, but 8-bit doesn't work for Moondream2 2B model.

Some initial model-related code from from my script:

import os
import re
import threading
from functools import partial
from concurrent.futures import ThreadPoolExecutor

import tkinter as tk
from tkinter import messagebox
from tkinterdnd2 import DND_FILES, TkinterDnD

import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

libvips_path = r".\Moondream\vips-dev-8.16\bin"
os.environ['PATH'] = libvips_path + os.pathsep + os.environ['PATH']

os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'

model = AutoModelForCausalLM.from_pretrained(
    "vikhyatk/moondream2",
    revision="2025-03-27",
    trust_remote_code=True,
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    torch_dtype=torch.float16,
    device_map={"": "cuda"})

def generate_caption(image_path, task_prompt):
    image = Image.open(image_path)
#    if image.mode != "RGB":
#        image = image.convert("RGB")
    
    length_mapping = {
        "Short_caption": "short",
        "Normal_caption": "normal",
        "Long_caption": "long"}
    
    with torch.no_grad():  # Disable gradient computation
        caption_result = model.caption(image, length=length_mapping[task_prompt])
    return caption_result["caption"]
...

Some installed pip packages and their versions:

Package                 Version
----------------------- ------------------------
accelerate              1.5.2
bitsandbytes            0.45.4
huggingface-hub         0.29.3
keras                   3.9.1
moondream               0.0.6
numpy                   2.1.2
opencv-contrib-python   4.11.0.86
opencv-python           4.11.0.86
opencv-python-headless  4.11.0.86
pillow                  11.1.0
protobuf                5.26.1
safetensors             0.5.3
tensorboard             2.19.0
tensorboard-data-server 0.7.2
tensorflow              2.19.0
termcolor               2.5.0
tf_keras                2.19.0
timm                    1.0.15
tkinterdnd2             0.4.3
tokenizers              0.21.0
torch                   2.8.0.dev20250327+cu128
torchvision             0.22.0.dev20250330+cu128
transformers            4.50.3

[manafpv]

HI bro i am also getting this issue did you correct this issue