ggml icon indicating copy to clipboard operation
ggml copied to clipboard

Published 20 hours ago • verified publisher icon ggml-org

Implement Quick GELU

Open monatis opened this issue 2 years ago • 3 comments

Closes #253

This is needed for CLIP-like models and I'm implementing clip.cpp here. It will also be a base for upcoming multimodal models that uses CLIP as an image embedder.

monatis avatar Jun 12 '23 22:06 monatis

you let some formatting happen. please only commit code changes.

Green-Sky avatar Jun 12 '23 23:06 Green-Sky

Oh sorry, missed that. I'll fix it.

monatis avatar Jun 12 '23 23:06 monatis

This is ready for review now.

monatis avatar Jun 13 '23 06:06 monatis

Hi @ggerganov Can ı have your attention here please? clip.cpp is almost ready, and I'm only crafting more examples and then I can announce it.

I can also add a link in the examples section in readme if you thing it deserves this.

p.s.: The next step will be using clip.cpp and llama.cpp to infer with LLaVA

monatis avatar Jun 16 '23 12:06 monatis

Unfortunately that theoretically small divergence between GELU and Quick GELU lead to large differences at the end, I suppose it accumulates through 12 layers. So I couldn't get good results until implementing Quick GELU.

monatis avatar Jun 16 '23 14:06 monatis