simple-llm-finetuner icon indicating copy to clipboard operation
simple-llm-finetuner copied to clipboard

Published 20 hours ago • verified publisher icon lxe

add cpu support

Open swap357 opened this issue 1 year ago • 3 comments

This pull request introduces the --use_cpu option to the main script, allowing users to run training and inference on the CPU. The --use_cpu option can be specified when running the script to enable CPU usage. By default, the script will use the GPU (if available).

Changes:

  • Added an argument parser for the --use_cpu option to specify CPU usage for training and inference.
  • Updated the load_base_model function to adjust load_in_8bit, torch_dtype, and device_map based on the use_cpu argument.
  • Modified the TrainingArguments to set fp16 based on the use_cpu argument.
  • Removed the .cuda() call from the generate_text function when using the CPU.

This update provides flexibility for users who want to run the script on systems without a GPU. The option to use the CPU can be helpful for testing, debugging, and using the script on machines with limited GPU resources.

swap357 avatar Apr 05 '23 06:04 swap357

Since I refactored the whole thing, you might need to adjust it a bit. Sorry

lxe avatar Apr 11 '23 04:04 lxe

The most expected pull request for now) to be able to work with llm on laptops

Alex-Pavlyuk avatar May 17 '23 05:05 Alex-Pavlyuk

I went through the latest code and seems like the updated code already takes care of running on cpu if cuda is unavailable. I believe the trainer should work out of the box on cpu now.

here: config.py - HAS_CUDA = torch.cuda.is_available() DEVICE = torch.device('cuda' if HAS_CUDA else 'cpu')

Let me know if otherwise.

swap357 avatar Jun 14 '23 04:06 swap357