starcoder
starcoder copied to clipboard
bigcode-project
Has anyone attempted to fine-tune the Starcoder model with your own code?
As per the title, I have attempted to fine-tune Starcoder with my own 400MB Python code. I concatenated all .py files into a single text file, similar to the content column of the bigcode/the-stack-dedup Parquet. I then scanned the text and sliced code snippets with 1024 characters to train the model for 1000 steps. However, during validation, I could not achieve any recall of my own code.
I have been informed that fine-tuning a language model does not require a large dataset.
May I know the manner you fine-tune your model and the way you estimate your performance of it, i.e. with fill-in-middle task or next-code-sentence prediction task? AFA I can know, the best fine-tuning performance can be obtained when your fine-tune following their instructions on Mega repository or fine-tune with fill-in-middle task given in GPT2.
However, if you are only validating your model in a model validation step and you find you code could not be reproduced, I suspect there could be implementation error with your code. You are welcome to provide your implementation snippet here so we could check.
