Llama 3.1 Load Fail

[DefinitlyEvil]

Output:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File ".../x/InfLLM/inf_llm/chat.py", line 726, in <module>
    main(args)
  File ".../x/InfLLM/inf_llm/chat.py", line 591, in main
    chat_loop(
  File ".../x/InfLLM/inf_llm/chat.py", line 347, in chat_loop
    model = patch_hf(model, inf_llm_config.type,  **inf_llm_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../x/InfLLM/inf_llm/utils/patch.py", line 156, in patch_hf
    hf_rope.dim,
    ^^^^^^^^^^^
  File ".../conda/envs/infllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'LlamaRotaryEmbedding' object has no attribute 'dim'

Even with Llama 3:

    model = patch_hf(model, inf_llm_config.type,  **inf_llm_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../InfLLM/inf_llm/utils/patch.py", line 156, in patch_hf
    hf_rope.dim,
    ^^^^^^^^^^^
  File ".../conda/envs/infllm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'LlamaRotaryEmbedding' object has no attribute 'dim'

[yuanyehome]

Hi, I got the same error. Did you fix it?

[DefinitlyEvil]

Hi, I got the same error. Did you fix it?

Yes, the API seemed changed, but it still didn't work well for me.

[yuanyehome]

Hi, I got the same error. Did you fix it?

Yes, the API seemed changed, but it still didn't work well for me.

I've changed the code to this:

It can run now. But I can't reproduce the baseline results(mistral-origin/llama3-origin) in the paper. It's really strange that I got a much lower score for the origin setting.

[DefinitlyEvil]

Hi, I got the same error. Did you fix it?

Yes, the API seemed changed, but it still didn't work well for me.

I've changed the code to this:

It can run now. But I can't reproduce the baseline results(mistral-origin/llama3-origin) in the paper. It's really strange that I got a much lower score for the origin setting.

Yes the performance is worse as I observed.