[Feature Request] Salesforce Codegen model

[siilats]

trafficstars

Hi, could you give us a hint on how to convert the Salesforce codegen model to ggml? The models are here https://github.com/salesforce/CodeGen and they allow Github copilot style code complete locally. Thank you!

[ggerganov]

Can you point me to the python implementation of the models?

[prof-schacht]

The python implementation should be here: https://github.com/salesforce/CodeGen/blob/main/jaxformer/hf/codegen/modeling_codegen.py

Could you explain how to convert these models first to ggml and then to cpp.

(P.S.: @ggerganov I contacted you also on twitter)

[Ayushk4]

I have fast CPP implementations of Codegen (at int4 quantization) at https://github.com/NolanoOrg/cformers/

from interface import AutoInference as AI
ai = AI('Salesforce/codegen-6B-mono')
x = ai.generate('def return1():, num_tokens_to_generate=50)
# print(x['token_str'])

It has same backend as codegen. Let me know if you need any help with it.

[prof-schacht]

Great I will try it out. Thanks for your hint.

[virtualramblas]

Working code from @ravenscroftj can be found here: https://github.com/ravenscroftj/ggml/tree/d130cc0c799621637c7914d6b89bf99e4403bd3a/examples/codegen I have written a more detailed tutorial about it to do conversion and quantization on Apple Silicon M1/M2 CPU: https://github.com/virtualramblas/codegen-quantization-M1 and shared some of the ported to ggml and quantized mono models through the Hugging Face Hub: https://huggingface.co/Guglielmo/CodeGen-350M-mono-ggml-quant https://huggingface.co/Guglielmo/CodeGen-2B-mono-ggml-quant https://huggingface.co/Guglielmo/CodeGen-6B-mono-ggml-quant