[WIP] Add FLMR model

[LinWeizheDragon]

What does this PR do?

This PR aims to add a new model, Fine-grained Late-interaction Multi-modal Retriever (FLMR). The model was proposed in here and here. This PR contains the following content:

• Model class for both FLMR and PreFLMR.
• Example scripts to run indexing and inference with FLMR and PreFLMR.

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline, Pull Request section?
• [ ] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [x] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR. @ArthurZucker @younesbelkada @amyeroberts

[LinWeizheDragon]

@ArthurZucker @younesbelkada @amyeroberts Can anyone help me to finish this PR or assign someone knowledgeable? The whole process is a little complicated and I have no idea what to do next. Thanks!

[ArthurZucker]

Hello @LinWeizheDragon, could you update the readme to have links to the pretrained checkpoints, the original codebase etc? I would recommend you to first start with a code on the hub + an issue on transformers to add support for this model. If it picks up the interest of the community 😉

[LinWeizheDragon]

Hello @LinWeizheDragon, could you update the readme to have links to the pretrained checkpoints, the original codebase etc? I would recommend you to first start with a code on the hub + an issue on transformers to add support for this model. If it picks up the interest of the community 😉

@ArthurZucker Thanks for your reply. I have added the requested info to the description of this PR. We are the authors of this model. We plan to release the pre-trained models here directly to provide easy access to multi-modal late-interaction models. I followed the docs here to finish a runnable version. The checkpoints have been converted and uploaded to the hub. I think the remaining things are having knowledgeable people to review the changes and pass all the tests. Could you please let me know what to do next?

[amyeroberts]

Hi @LinWeizheDragon, thanks for all the work opening this PR!

As Arthur mentioned, we suggest adding the model directly to the hub. This is the recommended way to add models and we try to have as much support as possible for enabling this. This way, the model will be available to use and findable on the hub immediately. Adding a model into the transformers repo introduces maintenance costs. As such, the bar for adding models is high and the review process can take a long time.

[LinWeizheDragon]

Hi @LinWeizheDragon, thanks for all the work opening this PR!

As Arthur mentioned, we suggest adding the model directly to the hub. This is the recommended way to add models and we try to have as much support as possible for enabling this. This way, the model will be available to use and findable on the hub immediately. Adding a model into the transformers repo introduces maintenance costs. As such, the bar for adding models is high and the review process can take a long time.

@amyeroberts Thanks for your reply. Given that the model weights are already on the Hub, what is the recommended way of sharing the model classes and definitions with users? Is there a way to allow users to do "AutoModel.from_pretrained" easily?

[amyeroberts]

@LinWeizheDragon Yes! :D Here's a guide on custom models and making them available on the hub: https://huggingface.co/docs/transformers/custom_models

If you want to use AutoModel.from_pretrained, you'll need to make sure to register your model. Here's an example of modeling code on the hub: https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat/blob/main/modeling_baichuan.py

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.