LLaVA
LLaVA copied to clipboard
haotian-liu
[Question] Possibility for Multi-image input?
Question
I really enjoyed reading this paper and I have played with it few days and came up to this question: LLava Architecture is capable of having more than one image input. I tried give two images as an input, but the inference result was not good. That is probably the model was not trained with multiple image input. Have you tried training LLaVA with dataset that has more than one input??
Hi @jsg921019
Thank you for your interest in our work. Due to the current way of training, we do not observe the model having very good capability referring to / comparing with multiple images. We are working on improving this aspect as well, stay tuned! You can also checkout some of the discussions here.
This is also of great interest to our group as well. We work with pathology images and it can take more than one image to describe a region of interest due to image size. In this case we don't need to compare images, but allow several images to represent one thing. This would be similar in concept to MIL modeling (https://github.com/Project-MONAI/tutorials/tree/main/pathology/multiple_instance_learning).
@codybum Thanks for explaining this interesting direction! I am curious about if there is any plan in your group in working in this direction? Would be happy to integrate that into LLaVA :)
I am also looking forward to seeing this feature soon. For instance, GPT-4V can take several images and find relationships between objects on different images. It's pretty cool and has a variety of use cases.
I have a strong interest in this topic I want to know if this is possible or not for the current version by customizing the code
@haotian-liu In my understanding GPT4v slices higher resolution images into 512x512 images plus one context image and then tokenizes + collates those tokens. Have you tried something like this with the latest LLaVA model by any chance? Is it something worth trying? My use case is simlar to @codybum 's where I need to pass in higher resolution images.
