RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

[PetroMaslov]

Hi!

I tried to use peft model with Seq2SeqTrainer, got the next error:

Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Code:

tokenizer = AutoTokenizer.from_pretrained(args.model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
        "google/flan-t5-large",
        use_cache=False if args.gradient_checkpointing else True, 
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q", "v"],
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_2_SEQ_LM"
)

model = get_peft_model(model, lora_config)


trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
    )

[JamesDConley]

I've been hitting this same issue whenever I try to use gradient checkpointing with Lora. Not sure what the underlying problem is but seems to be tied to that.

[younesbelkada]

Hi ! Can you call model.enable_input_require_grads() before get_peft_model ? You have to use the main branch of transformers though (pip install --upgrade git+https://github.com/huggingface/transformers)

[JamesDConley]

Hi ! Can you call model.enable_input_require_grads() before get_peft_model ? You have to use the main branch of transformers though (pip install --upgrade git+https://github.com/huggingface/transformers)

This worked for me, thanks!

[psinger]

@younesbelkada

Thanks, your suggestion fixes this issue. Unfortunately, if I additionally run torch.compile(model) I get this error again. Any idea?

[totaltube]

Same error if opening lora adapter for additional training. Also seq2seq model flan-t5

config = PeftConfig.from_pretrained(peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(model, peft_model_id).to(device)
model.print_trainable_parameters()

And model.print_trainable_parameters() will output 0 trainable parameters. model.enable_input_require_grads() before model = PeftModel.from_pretrained fixed errors, but output of print_trainable_parameters() still 0

[pacman100]

Hello, is_trainable=True is required to load the pretrained adapter and have them in trainable mode. Please pass PeftModel.from_pretrained(model, peft_model_id, is_trainable=True).to(device) and let us know if that solves the issue

[totaltube]

Hello, is_trainable=True is required to load the pretrained adapter and have them in trainable mode. Please pass PeftModel.from_pretrained(model, peft_model_id, is_trainable=True).to(device) and let us know if that solves the issue

Yes, it works with this parameter. Thank you! And no need to do enable_input_require_grads()

[duanyiqun]

@psinger
Same here, I have the same problem when additionally run compile command, have you found anything related to this?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[AayushSameerShah]

@younesbelkada Hell yeah, it works!

[ManuSinghYadav]

Hi ! Can you call model.enable_input_require_grads() before get_peft_model ? You have to use the main branch of transformers though (pip install --upgrade git+https://github.com/huggingface/transformers)

This worked for me, thanks!! @younesbelkada

[dengchengxifrank]

This problem may exit if you don't add the external model in to forward. Maybe you can check your forward function in your model.

[tuanlda78202]

This problem may exit if you don't add the external model in to forward. Maybe you can check your forward function in your model.

That's work for my case

[jacklanda]

Hi ! Can you call model.enable_input_require_grads() before get_peft_model ? You have to use the main branch of transformers though (pip install --upgrade git+https://github.com/huggingface/transformers)

This method is also workable for the scenario that you want to freeze some parts of the model and optimize the other parts on parameters. It'd be better to set the same thing for it.

[han-shanshan]

model.enable_input_require_grads()

Hi ! Can you call model.enable_input_require_grads() before get_peft_model ? You have to use the main branch of transformers though (pip install --upgrade git+https://github.com/huggingface/transformers)

You saved a soul.