model type `multi_modality` but Transformers does not recognize this architecture

[Huowuge]

trafficstars

Anyone encountered this error?

ValueError: The checkpoint you are trying to load has model type multi_modality but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

System: ubuntu 2204 python:3.10 transformers:latest torch:2.2.2 models: 1b and 1.3b

[gnusupport]

Same problem here

[gnusupport]

(TTS) lco@lco2:~/Programming/LLM/DeepSeek$ python Janus-Pro-1B.py 
Traceback (most recent call last):
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1073, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 775, in __getitem__
    raise KeyError(key)
KeyError: 'multi_modality'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/data1/protected/Programming/LLM/DeepSeek/Janus-Pro-1B.py", line 7, in <module>
    model = AutoModel.from_pretrained(model_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1075, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

for following attempt:

from transformers import AutoModel, AutoTokenizer
import torch
import torch.quantization

# Load the Janus-Pro-1B model and tokenizer
model_name = 'Janus-Pro-1B'  # Path to the Janus-Pro-1B directory
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Apply dynamic quantization to the model
# Quantize only the linear layers to reduce memory usage
quantized_model = torch.quantization.quantize_dynamic(
    model,  # The original model
    {torch.nn.Linear},  # Layers to quantize (e.g., linear layers)
    dtype=torch.qint8  # Quantization type (8-bit integers)
)

# Move the quantized model to the GPU (if available)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
quantized_model.to(device)

# Example input
input_text = "Hello, how are you?"
inputs = tokenizer(input_text, return_tensors='pt').to(device)  # Move inputs to the same device as the model

# Run the quantized model
with torch.no_grad():
    outputs = quantized_model(**inputs)

# Print the outputs
print(outputs)

[LeeABarron]

Same here with

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "deepseek-ai/Janus-Pro-7B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True)

[dineshchitlangia]

Same issue here with code similar to @gnusupport and @LeeABarron. I also tried the example inference.py file and that produces the same error too!

https://github.com/deepseek-ai/Janus/blob/1daa72fa409002d40931bd7b36a9280362469ead/inference.py

@hills-code @charlesCXK - could you pls help?

[YoshitoMurayama]

I think we have to register custom model classes to "transformers" before loading the model like line below https://github.com/deepseek-ai/Janus/blob/1daa72fa409002d40931bd7b36a9280362469ead/janus/models/modeling_vlm.py#L272

maybe importing janus.models execute above line

otherwise I solve this by adding custom model code to model repository(local folder), then I add "auto_map" settings to config.json...

[creasy2010]

Same problem

System: mac m1 python:3.8 transformers:latest torch:2.4.1 models: Janus-Pro-7B and Janus-Pro-1B and Janus-1.3B

[LeeABarron]

@YoshitoMurayama can you go into detail about your solution please? That would be great. Thanks man.

[DevPaulLiu]

Seems HuggingFace team are working on it: https://github.com/huggingface/transformers/issues/35928

[Wuwenji18]

同问

[popcornkiller1088]

this worked for me