[QUESTION] How to set `--rotary-seq-len-interpolation-factor` for rope scaling?

[eagle705]

Hi,

I'm curious if it's possible to set --rotary-seq-len-interpolation-factor in Megatron-LM to match huggingface's rope_scaling setting ({"type": "dynamic", "factor": 2.0}).

Is there any information you can share on how the --rotary-seq-len-interpolation-factor option is used?

[ethanhe42]

https://github.com/NVIDIA/Megatron-LM/blob/4bd4e7426c8def582eb80ae53f296a803643d2f5/megatron/core/models/common/embeddings/rotary_pos_embedding.py#L52

[github-actions[bot]]

Marking as stale. No activity in 60 days.