Fine-tuning Code Llama on my own code

[g12bftd]

Thank you for this amazing effort!

I would like to fine-tune code llama on my own Python code, let's call it MyPackage for now. Ultimately, I would like to ask Code llama questions of the form: "how do I do [INSERT FEASIBLE TASK NAME] using MyPackage?"

I have my package ready and documented. How do I format the dataset to fine-tune Code Llama? Are there fine-tuning scripts available in this repository?

[shatealaboxiaowang]

same issue.

[david-strejc]

Any answers for this question?

[jgehring]

Hi @g12bftd, we don’t host any fine-tuning scripts in this repository. You can check out https://github.com/facebookresearch/llama-recipes which includes fine-tuning recipes for Llama 2 models and works with Code Llama as well.

Wrt dataset formatting, this is hard to answer from the get-go. However, getting LLMs to work with data not seen in training is a frequent use case so I would expect you’d find various further resources on the web. I’d probably start with in-context learning (i.e., putting relevant or abbreviated contents of your repo in your prompt) instead of fine-tuning.