transformers
transformers copied to clipboard
Add ColPali to π€ transformers
What does this PR do?
Add ColPali support in π€ transformers.
Who can review?
@yonigozlan π
Additional details
- This PR uses the new Modular π€ transformers feature from v4.45.0
- The ColPali is mainly inspired from the repository I'm maintaining with my co-authors. The code is taken from
colpali-engine==v0.3.0.
Progress checklist
TODO
- [x] (Optional) Understood the modelβs theoretical aspects
- [x] Prepared π€ Transformers dev environment
- [x] Set up debugging environment of the original repository
- [x] Created script that successfully runs the forward() pass using the original repository and checkpoint
- [x] Successfully added the model skeleton to π€ Transformers
- [x] Successfully converted original checkpoint to π€ Transformers checkpoint
- [x] Successfully ran forward() pass in π€ Transformers that gives identical output to original checkpoint
- [ ] Finished model tests in π€ Transformers
- [x] Successfully added tokenizer in π€ Transformers
- [ ] Run end-to-end integration tests
- [ ] Finished docs
- [ ] Uploaded model weights to the Hub
- [ ] Submitted the pull request
- [ ] (Optional) Added a demo notebook