Dongfu Jiang

Also encounter this problem. Has been strugging with it for a long time...

Yes! Mantis used an image separator between images, but in a text form. We have prepended and to each placeholder in the text automatically. These codes are written in MllavaProcessor...

We actually wrote it in the paper. Every image token `` will be transformed into `(image {i}, ` where `` is the beginning of image token, and `` is the...

Thanks for the questions! During the training of idefics2, we disabled the image splitting (slicing) by default to make the image tokens more efficient. We also disable the image splitting...

@lucasjinreal Thanks for the questions! The first question sounds like a problem about continue-finetuning (continue-pretraining), Indeed, knowledge forgetting in continual training cannot be avoided and the most intuitive method to...

MiniCPM is a great baseline! We will include it as both a baseline and backbone in the future. We have some directions for Mantis' future work. We are actively working...

It's written in the model's processor file: See this [function](https://github.com/TIGER-AI-Lab/Mantis/blob/384b46082c9872a65690582adfc687c01e4efefb/mantis/models/mllava/processing_llava.py#L66)

For Mantis-Idefics2, it's literally the same as How Idefics2 is used. I found a existing [issue](https://github.com/vllm-project/vllm/issues/4124) in VLLM project that there are some people trying to implement Idefics2 in VLLM,...

also encouter same error when processing this image: [https://f2c628843e9892f5c7.gradio.live/file=/tmp/gradio/3036880890cf17b59a0cc838afc217dcd4d91ba5bc294ff42a99f6a2090f8bf2/equation.png] What's really weird is that, once I resize it to `(3844, 2408)`, then it will work. Error: ```txt [rank0]: Traceback (most...

I also tried to resize the image to `(1024, 1024)`, still error. Seems there will be error if the image is more like a square shape?