What kind of data to feed to the model ?

[dimwael]

What kind of data to provide for finetuning ? What are the best practices for finetuning ?

[Rartg]

The type of data required for fine-tuning depends on the task the model is intended to perform.

Best practices for fine-tuning include:

• Using a similar data distribution as the original data
• Using enough data to ensure the model can generalize well
• Using multiple epochs to allow the model to learn from the new data
• Using a small learning rate to prevent overfitting
• Monitoring the model's performance on a validation set
• Using a pre-trained model for better performance
• Regularizing the model to prevent overfitting

[csris]

If you're trying to reproduce the GPT-NeoXT-Chat-Base-20B model, you can download the dataset by running python data/OIG/prepare.py from the root of the repository.

We plan to add more documentation about fine tuning on your own data. Please see #10.