RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the sam

[AnaRhisT94]

Regarding blip2_feature_extraction.ipynb found in: https://github.com/salesforce/LAVIS/blob/main/examples/blip2_feature_extraction.ipynb

When running on CPU it works fine, but on GPU I'm getting this error: RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

I tried using model.half() and image = image.to(torch.float16) but it doesn't seem to work.

This happens in the following code: features_multimodal = model.extract_features(sample)

@dxli94 any ideas how to solve it?

[AnaRhisT94]

In load_model_and_preprocess I just made it: model = model.float() and commented if device == "cpu":

Will check later how to do use weights half() as well. If you have a solution until then please let me know.

[LiJunnan1992]

#169 should be able to fix this. Could you kindly help confirm?

[dxli94]

@AnaRhisT94 is it working now?

With https://github.com/salesforce/LAVIS/commit/baad2d7c8df599d8d9b081ba2e946626eaa2dc34, CPU support should also work.

[gulucaptain]

I also met this problem when using blip2_qformer model. I change the parameter 'vit_precision' with 'fp32'. This makes my code work. I hope it can help you.

[wildphoton]

I also met this issue. It is weird that the issue arises in some function but not others that all calls the same visual encoder self.ln_vision(self.visual_encoder(image)). Does anyone know why?