Model loaded from_pretrained Lora model goes OOM when trained.

[djaym7]

Training Model loaded from scratch - No OOM: pretrained= 'google/flan-t5-xl' model = AutoModelForSeq2SeqLM.from_pretrained(pretrained,low_cpu_mem_usage=True) tokenizer = AutoTokenizer.from_pretrained(pretrained) peft_config = LoraConfig( task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1) model.enable_input_require_grads() #https://github.com/huggingface/peft/issues/137 model = get_peft_model(model, peft_config)

Training Model loading from pre-trained - OOM: pretrained = 'saved_dir' config = PeftConfig.from_pretrained(pretrained) model = AutoModelForSeq2SeqLM.from_pretrained('google/flan-t5-xl',low_cpu_mem_usage=True) model = PeftModel.from_pretrained(model, pretrained) tokenizer = AutoTokenizer.from_pretrained('google/flan-t5-xl')

The model being loaded from pretrained is saved using : trainer.model.save_pretrained(pretrained)

I'm using deepspeed 3

[djaym7]

@dukesun99 @pacman100 is there something like inference_mode that has to be turned off or anything missing here which is causing this behavior ?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[Sanster]

@djaym7 hi, Did you solve this problem?

[younesbelkada]

It seems to be resolved by the latest update of bitsandbytes. Check out the release notes here: https://github.com/TimDettmers/bitsandbytes/releases/tag/0.39.1

pip install --upgrade bitsandbytes

Should do the trick

[djaym7]

will try it out, thanks

[Sanster]

I fixed the vRAM OOM problem by change map_location to cpu: https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L496

adapters_weights = torch.load(
    filename, map_location=torch.device("cpu")
)