open_clip icon indicating copy to clipboard operation
open_clip copied to clipboard

Make coca and HF work together

Open gpucce opened this issue 2 years ago • 1 comments

@ae86280 the HF support is indeed not good for coca yet this PR is to address this and also fix #445 @fedshyvana this will also remove the two unused args #434 while adding options for HF @vturrisi this is the PR where I plan to add the option to output the raw image tokens in CoCa #458

I should be able to work on it in the next days and fix this. So far there is nothing much.

High level Todos are:

  • [ ] add causal masking for roberta
  • [ ] add gpt2 as decoder
  • [ ] adjust configs
  • [ ] add option to output image tokens in coca visual encoder

gpucce avatar Feb 22 '23 18:02 gpucce

is there still a plan to support coca + HF models ?

jqyin avatar Jul 11 '23 16:07 jqyin