[bug] Generation Error: rope_deltas has not the same device of cache_position

[JJJYmmm]

Code snippet

https://github.com/huggingface/transformers/blob/11afab19c0e4b652855f9ed7f82aa010c4f14754/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L1792-L1800

Related issue

https://github.com/hiyouga/LLaMA-Factory/issues/6910

solution

modify line 1792 to self.rope_deltas = rope_deltas.to(cache_position.device)

    position_ids, rope_deltas = self.get_rope_index(
        input_ids,
        image_grid_thw,
        video_grid_thw,
        second_per_grid_ts,
        attention_mask,
    )
    self.rope_deltas = rope_deltas.to(cache_position.device)
# then use the prev pre-calculated rope-deltas to get the correct position ids
else:
    batch_size, seq_length, _ = inputs_embeds.shape
    delta = (
        (cache_position[0] + self.rope_deltas).to(inputs_embeds.device)
        if cache_position is not None
        else 0
    )