clip-as-service icon indicating copy to clipboard operation
clip-as-service copied to clipboard

feat: support 4 new clip models pretrained on laion2b

Open ZiniuYu opened this issue 2 years ago • 3 comments

This pr supports 4 new clip models introduced by https://laion.ai/blog/large-openclip/

  • ViT-B-32::laion2B-s34B-b79K (PyTorch+ONNX)
  • ViT-L-14::laion2B-s32B-b82K (PyTorch+ONNX)
  • ViT-H-14::laion2B-s32B-b79K (PyTorch)
  • ViT-g-14::laion2B-s12B-b42K (PyTorch)

It also sets timeout to 30 min in torch and onnx runtime since we have larger model files now

ZiniuYu avatar Sep 20 '22 16:09 ZiniuYu

Codecov Report

Merging #825 (3835f75) into main (c690c24) will decrease coverage by 2.09%. The diff coverage is 62.50%.

@@            Coverage Diff             @@
##             main     #825      +/-   ##
==========================================
- Coverage   84.67%   82.57%   -2.10%     
==========================================
  Files          21       21              
  Lines        1566     1567       +1     
==========================================
- Hits         1326     1294      -32     
- Misses        240      273      +33     
Flag Coverage Δ
cas 82.57% <62.50%> (-2.10%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/clip_server/model/clip_onnx.py 88.88% <ø> (ø)
server/clip_server/model/pretrained_models.py 98.41% <ø> (ø)
server/clip_server/model/model.py 78.97% <62.50%> (+0.04%) :arrow_up:
server/clip_server/model/trt_utils.py 56.04% <0.00%> (-27.48%) :arrow_down:
server/clip_server/model/clip_trt.py 69.38% <0.00%> (-16.33%) :arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov[bot] avatar Sep 20 '22 17:09 codecov[bot]

:memo: Docs are deployed on https://ft-large-model--jina-docs.netlify.app :tada:

github-actions[bot] avatar Sep 21 '22 09:09 github-actions[bot]

server/clip_server/model/model.py

This file is synced with https://github.com/mlfoundations/open_clip/blob/main/src/open_clip/model.py

ZiniuYu avatar Sep 21 '22 10:09 ZiniuYu

@ZiniuYu Please polish the PR's title. 4 new clip models is meaningless. Please highlights the contribution in the title

numb3r3 avatar Sep 23 '22 07:09 numb3r3