llm-course icon indicating copy to clipboard operation
llm-course copied to clipboard

Update Fine-tune Llama 2 libraries

Open appleparan opened this issue 1 year ago • 0 comments

To enable kbit quantization, gradient_checkpointing must be passed into TrainingArguments

  1. I remove library version restriction.
!pip install -q accelerate peft bitsandbytes transformers trl[quantization]
  1. Also, I pass gradient_checkpointing into TrainingArguments. (trl>=0.7.2)

# Set training parameters
training_arguments = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=num_train_epochs,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    gradient_checkpointing=gradient_checkpointing,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    weight_decay=weight_decay,
    fp16=fp16,
    bf16=bf16,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    group_by_length=group_by_length,
    lr_scheduler_type=lr_scheduler_type,
    report_to="tensorboard"
)

appleparan avatar Jan 22 '24 15:01 appleparan