open_clip
open_clip copied to clipboard
Make coca and HF work together
@ae86280 the HF support is indeed not good for coca yet this PR is to address this and also fix #445 @fedshyvana this will also remove the two unused args #434 while adding options for HF @vturrisi this is the PR where I plan to add the option to output the raw image tokens in CoCa #458
I should be able to work on it in the next days and fix this. So far there is nothing much.
High level Todos are:
- [ ] add causal masking for roberta
- [ ] add gpt2 as decoder
- [ ] adjust configs
- [ ] add option to output image tokens in coca visual encoder
is there still a plan to support coca + HF models ?