BLIP2 models would stop working with next release of tranformers (4.27.0)

[AstraliteHeart]

I've been trying to use transformers to convert BLIP2 models into compatible format (https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/convert_blip_2_original_to_pytorch.py) and ran into some issues, specifically this commit https://github.com/huggingface/transformers/pull/21405 breaks compatibility between lavis and transformers.

So while loading models via transformers (i.e. https://huggingface.co/docs/transformers/main/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example) will continue to work, lavis will need to be updated for compatibility with next release.

Reference issue: https://github.com/huggingface/transformers/issues/21713