Error when use tag2text in batch inference

[Kailuo-Lai]

First of all, I would like to express my gratitude for your work, the tag2text model has helped me a lot. But when I try to use this model in batch inference, I encounter an unexpected error. My code as follows:

1.

I download 3 images and then I follow the instruction to transform images into proper format.

frame_links = ['http:...', 'http://...', 'http:...']
images = [model.transform(Image.open(BytesIO(requests.get(url = x).content))).unsqueeze(0) for x in frame_links]
images = torch.concat(images)

output

# (N, C, H, W)
torch.Size([3, 3, 384, 384])

2.

with torch.inference_mode():
    test = model.model.generate(torch.concat(images),
                                tag_input=None,
                                max_length=50,
                                return_tag_predict=True)

output

(['a young girl dressed in a blue dress and hat',
  'a young girl dressed in a blue dress and hat',
  'a young girl dressed in a blue dress and hat'],
 ['hat | costume | girl | dress | person | dress | blue | young | little',
  'hair | doll | room | woman | girl | game | toy | person | pink | blue | white',
  'smile | football player | room | chair | person | man | laugh | smile | wear'])

It seems that the caption part works incorrectly. It just repeats the caption result of the first image. At the same time, the tag part returns correct result. I think maybe the problem occurs in the generate function. Can you help me to solve this bug?

[xinyu1205]

Hi, thank you for your valuable feedback! Can you try lowering the Transformers version to 4.15.0 and review the results? Looking forward to your reply. pip install transformers==4.15.0

[xinyu1205]

If reducing the version of Transformers does not solve the problem, please feel free to reply.

[Kailuo-Lai]

Sorry for my delay. Thanks for your advice, it has successfully solved the problem. The previous version of transformers in my pc was 4.33.2, hope this information will help you modify the code.

[xinyu1205]

I'm very glad this solved your error. If convenient, could you please check if the results when transformers=4.25.0 is also correct? This will have significant value for updating this code base. pip install transformers==4.25.0

[Kailuo-Lai]

Yes, it also works when transformers=4.25.0. Thank you.