Img2txt result is pretty bad on 16bit

[billzhao9]

Don't know if I did anything wrong, but the result was not like the example.

import os from core.models.model_module_infer import model_module

model_load_paths = ['CoDi_encoders.pth', 'CoDi_text_diffuser.pth', 'CoDi_audio_diffuser_m.pth', 'CoDi_video_diffuser_8frames.pth'] inference_tester = model_module(data_dir='checkpoints/', pth=model_load_paths, fp16=True) # turn on fp16=True if loading fp16 weights inference_tester = inference_tester.cuda() inference_tester = inference_tester.eval()

from PIL import Image im = Image.open('./assets/demo_files/house.jpeg').resize((224,224)) im text = inference_tester.inference( xtype = ['text'], condition = [im], condition_types = ['image'], n_samples = 4, ddim_steps = 50, scale = 7.5,) text[0] Data shape for DDIM sampling is [[4, 768]], eta 0.0 DDIM Sampler: 100%|██████████| 50/50 [00:01<00:00, 29.62it/s] ['oriental book examines a bag with bright spots.', 'a street view of kitchen toys and tv.', 'a is also carrying a ton of blue and green spandex around and pedals.', 'woman and a white bird crossing in the field.']

The sample image is a house.. but somehow the result is very strange.

[zinengtang]

What is your transformers version? Did you install requirements.txt

[billzhao9]

Yes. I have all modules installed as requirements.txt listed. However, the transformers version is 4.33.2. Did it impact the result?

[zinengtang]

I think so. The transformers version in requirements.txt is 4.26.0. The higher version can cause some mismatch to the code.

[jacklishufan]

@billzhao9 I might have similar issues with Image Encoder. Were you able to fix the issue by fixing transformer version?