TypeError: WhisperForConditionalGeneration.forward() got an unexpected keyword argument 'input_ids'

[YOUSEFNANIS]

System Info

peft version = '0.12.0' transformers version = '4.41.2' accelerate version = '0.30.1' bitsandbytes version = '0.43.3' pytorch version = '2.1.2'

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder
• [ ] My own task or dataset (give details below)

Reproduction

import torch from transformers import WhisperForConditionalGeneration from peft import get_peft_model, LoraConfig, TaskType, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(load_in_8bit=True) whisper_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny", quantization_config = bnb_config, device_map='auto') quantized_model = prepare_model_for_kbit_training(whisper_model )

peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, target_modules=["q_proj", "v_proj"], r=32, lora_alpha=64, lora_dropout=0.1) final_model = get_peft_model(quantized_model , peft_config)

final_model(torch.zeros([1, 80, 3000])) # torch.zeros is just used as a dummy input

Expected behavior

This TypeError that appears to be caused by recent changes in the peft library. I found similar codes run fine but whenever I try to run the same code it rises this Error. A potential workaround is to create a custom WhisperForConditionGeneration subclass, modifying the forward method to accept "input_ids" and rename it to "input_features" inside the body of forward method. However, this solution seems inefficient.

[phamvanhoang9]

@YOUSEFNANIS - Remove task_type from LoraConfig, which helps me resolve this issue. You can find a solution at: https://discuss.huggingface.co/t/unexpected-keywork-argument/91356

[BenjaminBossan]

Thanks @phamvanhoang9 for suggesting a solution. @YOUSEFNANIS let us know if this solves the problem for you or not.

[YOUSEFNANIS]

@YOUSEFNANIS - Remove task_type from LoraConfig, which helps me resolve this issue. You can find a solution at: https://discuss.huggingface.co/t/unexpected-keywork-argument/91356

thanks, it worked.

[BenjaminBossan]

Great, thanks for reporting back. Feel free to re-open the issue if something new comes up.

[tolgadur]

This is still a bug no? task_type should not affect a downstream input_ids keyword argument that is being passed. Sure, omitting it is a workaround but shouldn't we fix this bug instead of closing this ticket?

[githubnemo]

@tolgadur the task type definitely changes semantics for the model inputs. The same goes for model outputs. You wouldn't argue against a classification task having different outputs than a language modeling task, right?

[tolgadur]

@githubnemo OP used task_type=TaskType.SEQ_2_SEQ_LM, to fine-tune whisper for a sequence-to-sequence task, same as I have done. If the behaviour is correct then adding the correct task I am fine tuning whisper for should not output an error saying "unexpected keyword argument 'input_ids'".

What happens under the hood is if you omit task_type, peft returns the base PeftModel for which fine-tuning works, however for the PeftModelForSeq2SeqLM it gives me the input_ids error. How is this not a bug? I am trying to do seq-2-seq fine tuning here, aren't I?

MODEL_TYPE_TO_PEFT_MODEL_MAPPING: dict[str, type[PeftModel]] = {
    "SEQ_CLS": PeftModelForSequenceClassification,
    "SEQ_2_SEQ_LM": PeftModelForSeq2SeqLM,
    "CAUSAL_LM": PeftModelForCausalLM,
    "TOKEN_CLS": PeftModelForTokenClassification,
    "QUESTION_ANS": PeftModelForQuestionAnswering,
    "FEATURE_EXTRACTION": PeftModelForFeatureExtraction,
}

if peft_config.task_type not in MODEL_TYPE_TO_PEFT_MODEL_MAPPING.keys() and not peft_config.is_prompt_learning:
        return PeftModel(
            model,
            peft_config,
            adapter_name=adapter_name,
            autocast_adapter_dtype=autocast_adapter_dtype,
            low_cpu_mem_usage=low_cpu_mem_usage,
        )

    if peft_config.is_prompt_learning:
        peft_config = _prepare_prompt_learning_config(peft_config, model_config)
    return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
        model,
        peft_config,
        adapter_name=adapter_name,
        autocast_adapter_dtype=autocast_adapter_dtype,
        low_cpu_mem_usage=low_cpu_mem_usage,
    )

[githubnemo]

@githubnemo OP used task_type=TaskType.SEQ_2_SEQ_LM, to fine-tune whisper for a sequence-to-sequence task, same as I have done. If the behaviour is correct then adding the correct task I am fine tuning whisper for should not output an error saying "unexpected keyword argument 'input_ids'".

What happens under the hood is if you omit task_type, peft returns the base PeftModel for which fine-tuning works, however for the PeftModelForSeq2SeqLM it gives me the input_ids error. How is this not a bug? I am trying to do seq-2-seq fine tuning here, aren't I?

I understand, let me clarify. The task type is TaskType.SEQ_2_SEQ_LM, i.e. sequence-to-sequence language modelling. If you'd use WhisperForCausalLM and it wouldn't work, I would agree that this is a bug. However, since we're talking about language modelling, the PeftModelForCausalLM (the PEFT class behind TaskType.SEQ_2_SEQ_LM) assumes that there has to be an input_ids argument - because we're language modelling.

There's currently no specific task for a model like Whisper. It is a sequence-to-sequence model but when using WhisperForConditionalGeneration it is not a language model. That's the reason why using a task type will lead to this error.

[cmb-code]

I get this error even while running inference.

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from transformers.pipelines import pipeline


device = "mps"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch_dtype,
    device=device,
)

sample = "slow.wav"

generate_kwargs = {
    "num_beams": 1,
    "condition_on_prev_tokens": False,
    "compression_ratio_threshold": 1.35,  # zlib compression ratio threshold (in token space)
    "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
    "logprob_threshold": -0.5,
    "no_speech_threshold": 0.6,
    "return_timestamps": True,
}

result = pipe(sample, generate_kwargs=generate_kwargs)

print(result)

[githubnemo]

@cmb-code I don't think this has anything to do with PEFT. If you remove the no_speech_threshold option the example will work. I'm not familiar enough with the Whisper implementation to decide if it is a bug or just surprising but if it is not easily resolved, I'd suggest that you raise a bug in transformers.

[Technologicat]

@githubnemo Thank you. Indeed nothing to do with PEFT. I got this during inference too, and removing no_speech_threshold helped.