GPT-Neo-2.7B & GPT-J-6B Fine-Tuning Examples Using HuggingFace & DeepSpeed

Installation

cd venv/bin
./pip install -r ../../requirements.txt 
./pip install deepspeed==0.5.9

GPT-Neo

Example with GPT-Neo-1.3B without DeepSpeed
Training and testing log with GPT-Neo-1.3B
Example with GPT-Neo-2.7B with DeepSpeed
DeepSpeed configuration with GPT-Neo-2.7B
Training and testing log with GPT-Neo-2.7B

GPU VRAM load during GPT-Neo-2.7B training

RAM load during GPT-Neo-2.7B training

Results

GPT-J-6B

Example with GPT-J-6B with DeepSpeed
DeepSpeed configuration with GPT-J-6B
Training and testing log with GPT-J-6B

GPU VRAM load during GPT-J-6B training

RAM load during GPT-J-6B training