llm-course
llm-course copied to clipboard
Update Fine-tune Llama 2 libraries
To enable kbit quantization, gradient_checkpointing must be passed into TrainingArguments
- I remove library version restriction.
!pip install -q accelerate peft bitsandbytes transformers trl[quantization]
- Also, I pass
gradient_checkpointingintoTrainingArguments. (trl>=0.7.2)
# Set training parameters
training_arguments = TrainingArguments(
output_dir=output_dir,
num_train_epochs=num_train_epochs,
per_device_train_batch_size=per_device_train_batch_size,
gradient_accumulation_steps=gradient_accumulation_steps,
gradient_checkpointing=gradient_checkpointing,
optim=optim,
save_steps=save_steps,
logging_steps=logging_steps,
learning_rate=learning_rate,
weight_decay=weight_decay,
fp16=fp16,
bf16=bf16,
max_grad_norm=max_grad_norm,
max_steps=max_steps,
warmup_ratio=warmup_ratio,
group_by_length=group_by_length,
lr_scheduler_type=lr_scheduler_type,
report_to="tensorboard"
)