Finetuning LLama 2 Code Instruct without using Hugging Face

[sgupta1007]

I tried to finetune CodeLlama 7b Instruct by downloading weights through official repository

Folder Structure of Folder Containing CodeLlama Instruct Weights

params.json contents

Torch Tune Command in google colab to Run FineTuning Job Using qLoRa tune run lora_finetune_single_device --config llama2/7B_qlora_single_device checkpointer.checkpoint_dir=/llama/CodeLlama-7b-Instruct/ tokenizer.path=lama/CodeLlama-7b-Instruct/tokenizer.model checkpointer.output_dir=/content/drive/llama/ checkpointer.checkpoint_files=[consolidated.00.pth] dtype=fp 32

Issues Faced 1 num_attention_heads not found when i rename params.json to config.json 2 For unaltered codellama weighs folder i faced .config.json not found issue

How to address num_attention_heads issue ?

[kartikayk]

@sgupta1007 Thanks for filing this issue! Good question. So there are a couple of things going on here:

• Checkpoint Format. Since you've downloaded from the original Meta repo, you'll need to update the checkpointer to the one that is able to understand this format. This is because the default checkpointer in the config understands the HF formatted checkpoint. You can do this by adding checkpointer=torchtune.utils.FullModelMetaCheckpointer to the command you have above. Take a look at the checkpointer deepdive for more information on checkpoint formats and how the checkpointer handles them.

• Change in Vocab Size. Even if you make the change above, you'll hit into the issue where the checkpoint weights dont match the shape of the model. This is because the base llama2 7B model supports a vocab_size of 32000 while code-llama 7B has a vocab_size of 32016. This should be a pretty simple change to add the code llama 7B model builders similar to this and this. Note that the max_seq_len for code-llama is also higher - 16384 compared to the 4096 of the base model. If you'd be up for making the change, I'd be happy to review the code. Or I can add in a PR on Monday morning.

Hope this helps!

[SalmanMohammadi]

I picked this up, hope it's not poor ettiquette :) https://github.com/pytorch/torchtune/pull/847

[SalmanMohammadi]

Happy to say this is implemented @sgupta1007 :) Feel free to try and update if there's any issues, otherwise, would it be appropriate to close this issue?

[sgupta1007]

@SalmanMohammadi I am still getting the error

Torchtune command and error are given below

Torchtune modified Command

!tune run lora_finetune_single_device --config llama2/7B_qlora_single_device checkpointer=torchtune.utils.FullModelMetaCheckpointer checkpointer.checkpoint_dir=/codellama-main/llama/CodeLlama-7b-Instruct/ tokenizer.path=/codellama-main/llama/CodeLlama-7b-Instruct/tokenizer.model checkpointer.output_dir=/llm/ checkpointer.checkpoint_files=[consolidated.00.pth] dtype=fp32 model=torchtune.models.llama2.code_llama2_7b

Error

ModuleNotFoundError("No module named 'torchtune.models.llama2.code_llama2_7b'")
Are you sure that 'code_llama2_7b' is importable from module 'torchtune.models.llama2'?

[SalmanMohammadi]

Could you try the following command?

!tune run lora_finetune_single_device --config code_llama2/7B_qlora_single_device checkpointer=torchtune.utils.FullModelMetaCheckpointer checkpointer.checkpoint_dir=/codellama-main/llama/CodeLlama-7b-Instruct/ tokenizer.path=/codellama-main/llama/CodeLlama-7b-Instruct/tokenizer.model checkpointer.output_dir=/llm/ checkpointer.checkpoint_files=[consolidated.00.pth] dtype=fp32

code_llama2 models and recipe configs are under their own folder, rather than under llama2/. I hadn't tried it with the ...MetaCheckpointer but hopefully it works. See my colab here for my examples.

[sgupta1007]

@SalmanMohammadi This command gave me new eror FileNotFoundError: [Errno 2] No such file or directory: '/content/code_llama2/7B_qlora_single_device'

[SalmanMohammadi]

My apologies, code_llama2 isn't in the release yet! You can follow the process here to install from source to get it working. Let me know how you get on.