Support for transformers>=4.50.0?

[deep1401]

Tried the tutorials with the transformers==4.50.0 version on Colab and the Platypus2-7B model throws the following error:

---> 24 generation_output = model.generate(
     25     input_tokens['input_ids'].cuda(),
     26     max_new_tokens=3,

7 frames
[/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py](https://localhost:8080/#) in decorate_context(*args, **kwargs)
    114     def decorate_context(*args, **kwargs):
    115         with ctx_factory():
--> 116             return func(*args, **kwargs)
    117 
    118     return decorate_context

[/usr/local/lib/python3.11/dist-packages/transformers/generation/utils.py](https://localhost:8080/#) in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, use_model_defaults, **kwargs)
   2324 
   2325             # 12. run sample (it degenerates to greedy search when `generation_config.do_sample=False`)
-> 2326             result = self._sample(
   2327                 input_ids,
   2328                 logits_processor=prepared_logits_processor,

[/usr/local/lib/python3.11/dist-packages/transformers/generation/utils.py](https://localhost:8080/#) in _sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
   3284 
   3285             if is_prefill:
-> 3286                 outputs = self(**model_inputs, return_dict=True)
   3287                 is_prefill = False
   3288             else:

[/usr/local/lib/python3.11/dist-packages/airllm/airllm_base.py](https://localhost:8080/#) in __call__(self, *args, **kwargs)
    367 
    368     def __call__(self, *args, **kwargs):
--> 369         return self.forward(*args, **kwargs)
    370 
    371     def get_past_key_values_cache_seq_len(self, past_key_values):

[/usr/local/lib/python3.11/dist-packages/airllm/airllm_base.py](https://localhost:8080/#) in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
    467                     if self.profiling_mode:
    468                         t = time.time()
--> 469                     moved_layers = self.move_layer_to_device(state_dict)
    470                     if self.profiling_mode:
    471                         elapsed_time = time.time() - t

[/usr/local/lib/python3.11/dist-packages/airllm/airllm_base.py](https://localhost:8080/#) in move_layer_to_device(self, state_dict)
    315                 not self.hf_quantizer.check_quantized_param(self.model, param_value=None, param_name=param_name, state_dict={})
    316                ):
--> 317                 set_module_tensor_to_device(self.model, param_name, self.running_device, value=state_dict[param_name],
    318                                             dtype=self.running_dtype,
    319                                             )

[/usr/local/lib/python3.11/dist-packages/accelerate/utils/modeling.py](https://localhost:8080/#) in set_module_tensor_to_device(module, tensor_name, device, value, dtype, fp16_statistics, tied_params_map)
    251         splits = tensor_name.split(".")
    252         for split in splits[:-1]:
--> 253             new_module = getattr(module, split)
    254             if new_module is None:
    255                 raise ValueError(f"{module} has no attribute {split}.")

[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in __getattr__(self, name)
   1926             if name in modules:
   1927                 return modules[name]
-> 1928         raise AttributeError(
   1929             f"'{type(self).__name__}' object has no attribute '{name}'"
   1930         )

AttributeError: 'LlamaAttention' object has no attribute 'rotary_emb'

Running the mistralai/Mistral-7B-Instruct-v0.1 gives the following error:

[/usr/local/lib/python3.11/dist-packages/transformers/models/mistral/modeling_mistral.py](https://localhost:8080/#) in forward(self, hidden_states, position_embeddings, attention_mask, past_key_value, cache_position, **kwargs)
    164         value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
    165 
--> 166         cos, sin = position_embeddings
    167         query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
    168 

TypeError: cannot unpack non-iterable NoneType object

[vishnupriyanpr]

Hey ! @deep1401 & @lyogavin I've investigated this issue and I have a clear idea for a fix. I'd like to volunteer to work on it. Could you please assign this to me? I'll get started on a PR.

Thanks !