Hardware requirements for inference and fine tuning.

[Vipitis]

Hey,

I am finishing a project on evaluating code language models on "creative" programming (shadercode). Just yesterday I finished fine-tuning sanatacoder on three different datasets to evaluate on my metric.

Now this new project popped up but it's vastly larger. Prohibitively so. I was unable to run 6B models on the RTX A5000 I have access to. For finetuning santacoder (no_fp16, batch_size 2 and sequence length of 2048) 97% of the 24GB VRAM was used using a slightly adapted version of the provided script.

Do you have any numbers on what requirements there are for PEFT on this model? And is there a set way to run inference with the hardware I listed?

Thanks for releasing yet another interesting model!

[Vipitis]

found #6 as closed which gives some information about inference.

Perhaps add that information to the readme

[loubnabnl]

Thanks for the feedback, this PR adds them

[ywen666]

Thanks for the hardware requirement info! It seems only have the requirement for inference. I wonder if there is anything for fine-tuning on downsteam tasks?