Phi4 MM will not perform text generation tasks

[palladium123]

Hi all

I tried using the code from the coobook for text inference with HF as found here:

https://github.com/microsoft/PhiCookBook/blob/main/md/01.Introduction/02/01.HF.md

When the model is set to Phi4- Mini- Instruct things work fine, however I get the below error if I use Phi4-multimodal-instruct:

Exception has occurred: ValueError None is not a valid InputMode File "C:\[redacted]\huggingface\modules\transformers_modules\Phi-4-multimodal-instruct\modeling_phi4mm.py", line 2101, in forward input_mode = InputMode(input_mode) ^^^^^^^^^^^^^^^^^^^^^ File "\phi4 text.py", line 45, in <module> output = pipe(messages, **generation_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: None is not a valid InputMode

For completeness, this is the code that generated the above error:

import torch
from transformers import AutoTokenizer,pipeline,AutoModelForCausalLM,BitsAndBytesConfig

model_path = ""

nf4_config = BitsAndBytesConfig(
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="cuda",
    attn_implementation="flash_attention_2",
    torch_dtype="auto",
    trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

messages = [
    {"role": "user", "content": "I have $20,000 in my savings account, where I receive a 4% profit per year and payments twice a year. Can you please tell me how long it will take for me to become a millionaire? Also, can you please explain the math step by step as if you were explaining it to an uneducated person?"},
]

pipe = pipeline(
    'text-generation',
    model=model,
    tokenizer=tokenizer,
)

generation_args = {
    "max_new_tokens": 4096,
    "return_full_text": False,
    "temperature": 0.00001,
    "top_p": 1.0,
    "do_sample": True,
}

output = pipe(messages, **generation_args)

print(output[0]['generated_text'])

Some help would be greatly appreciated.

[leestott]

If you have inference errors, please double check the environ and dependencies, this is what we suggest https://huggingface.co/microsoft/Phi-4-multimodal-instruct#requirements

Alternatively you can look at this dockerfile https://github.com/anastasiosyal/phi4-multimodal-instruct-server/blob/main/dockerfile