[Multimodal] General questions

[chengyjonathan]

Hi there, love the work here! I have a few questions about multimodal features.

Let's say I want to do some text -> image + text retrieval. Along the lines of the "Q & A over LlamaIndex Documentation"(https://github.com/jerryjliu/llama_index/blob/main/examples/multimodal/Multimodal.ipynb).

But let's say my images contain entities that are not recognized by clip, blip, etc. So it would be hard to support queries like "show me images of X-enitity?"

Do you have any recommendations on how to proceed?

1. Would adding information, along with other context, in the filename help support these kinds of queries? In the example, we can add metadata using the filename, but it isn't immediately clear to me if that metadata is encoded along with the image.

2. Is it possible to use our own fine tuned versions of blip or clip to embed our documents/queries? Ie. Along the lines of: (https://gpt-index.readthedocs.io/en/latest/how_to/embeddings.html).)

3. Other ideas?

Thank you!