KeyError: 'jetmoe' for jetmoe-8b-chat

[Sukii]

jetmoe-8b model runs fine but for jetmoe-8b-chat with even the latest transformers and tokenizer I get:

Traceback (most recent call last):
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 932, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 634, in __getitem__
    raise KeyError(key)
KeyError: 'jetmoe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cqrl/jetmoe/chat-jet.py", line 6, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, attn_implementation="eager", trust_remote_code=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 523, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 934, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `jetmoe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[Sukii]

>>> print(transformers.__version__)
4.40.0.dev0

[inuwamobarak]

I have the same issue. I have installed TF from git:

Attempting uninstall: transformers
    Found existing installation: transformers 4.38.2
    Uninstalling transformers-4.38.2:
      Successfully uninstalled transformers-4.38.2
Successfully installed transformers-4.40.0.dev0

I have also successfully install jetmoe:

Running setup.py develop for jetmoe
Successfully installed jetmoe-0.0.0 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.1.105 scattermoe-0.0.0

Yet I get the below error:

KeyError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs) 931 try: --> 932 config_class = CONFIG_MAPPING[config_dict["model_type"]] 933 except KeyError:

3 frames KeyError: 'jetmoe'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs) 932 config_class = CONFIG_MAPPING[config_dict["model_type"]] 933 except KeyError: --> 934 raise ValueError( 935 f"The checkpoint you are trying to load has model type {config_dict['model_type']} " 936 "but Transformers does not recognize this architecture. This could be because of an "

ValueError: The checkpoint you are trying to load has model type jetmoe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[inuwamobarak]

jetmoe-8b model runs fine but for jetmoe-8b-chat with even the latest transformers and tokenizer I get:

Traceback (most recent call last):
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 932, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 634, in __getitem__
    raise KeyError(key)
KeyError: 'jetmoe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cqrl/jetmoe/chat-jet.py", line 6, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, attn_implementation="eager", trust_remote_code=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 523, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cqrl/.local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 934, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `jetmoe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

But I wasn't able to run any of the models even jetmoe/jetmoe-8b or the chat.

[Sukii]

Please try pulling manually the model files directly from here to see if it works:

...
wget https://huggingface.co/jetmoe/jetmoe-8b-chat/resolve/main/tokenizer.model
...

[inuwamobarak]

Please try pulling manually the model files directly from here to see if it works:

...
wget https://huggingface.co/jetmoe/jetmoe-8b-chat/resolve/main/tokenizer.model
...

Can you please provide a little hint on how you would pull manually? I only have experience pulling GGUF models and loading them manually.

[Sukii]

Created a github repo for that: https://github.com/Sukii/jetmoe

Managed to run jetmoe-8b-chat also (replace AutoModelForCasualLM by JetMoEForCasualLM).

The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00,  1.30it/s]
GPU is not available, using CPU instead.
tensor([[  523, 28766,  6574, 28766, 28767,    13,  1976,   460,   264, 10131,
         10706, 10093,     2, 28705,    13, 28789, 28766,  1838, 28766, 28767,
            13,  5660,  1287, 19624,   410,  1532,   541,   264,  2930,  5310,
           297,   624,  6398, 28804,     2, 28705,    13, 28789, 28766,   489,
         11143, 28766, 28767,    13]])
<|system|>
You are a friendly chatbot 
<|user|>
How many helicopters can a human eat in one sitting? 
<|assistant|>
I'm sorry, but I can't provide that information. Helicopers are not a real thing and it's not possible for a person to eat a helicopter.

[Sukii]

Added pulling bash scripts in the model folders. Please move your discussions to the github repositary: https://github.com/Sukii/jetmoe