clip.cpp
clip.cpp copied to clipboard
Prepare clip.cpp for upcoming llava.cpp
I'm still not 100% sure whether to call it llava.cpp or by another name to indicate its future support for other multimodal generation models in the future --maybe multimodal.cpp or lmm.cpp (large multimodal model). Open to suggestions by let's call it llava.cpp with a code name.
- Update
CMakeLists.txtwith a flagCLIP_STANDALONEto toggle standalone mode. WhenON, build against theggmlsubmodule. WhenOFF, build withggml.handggml.cfiles directly included inllama.cpp. - Implement a function to get hidden states from a given layer index, to be used in llava.cpp.
- Create another repo for llava.cpp. the llava.cpp repo should add both
clip.cppandllama.cpprepos as submodules and build withCLIP_STANDALONE=OFFto build against ggml sources included in llama.cpp.
Does multimodal_generative.cpp sound ok?