ValueError: Transformers now supports natively BetterTransformer optimizations

[huangjf11]

ValueError: Transformers now supports natively BetterTransformer optimizations (torch.nn.functional.scaled_dot_product_attention) for the model type llama. As such, there is no need to use model.to_bettertransformers() or BetterTransformer.transform(model) from the Optimum library. Please upgrade to transformers>=4.36 and torch>=2.1.1 to use it.

I updated the version of transformers to 4.36 and torch to 2.1.1. However, I am still encountering this error.

[HamidShojanazeri]

Thanks @huangjf11 for opening this issue, is this on inference? its a legacy code in chat_example that need to be removed, code been updated here and in finetuning as well.

This PR fix the issue in chat_example. Also if you are facing it in the fine-tuning as you might have pip installed, pls install from src.

[devon-research]

I also encountered this error running

torchrun \
    --nnodes 1 \
    --nproc_per_node 4 \
    llama-recipes/examples/finetuning.py \
        --enable_fsdp \
        --model_name meta-llama/Llama-2-7b-hf \
        --dist_checkpoint_root_folder model_checkpoints \
        --dist_checkpoint_folder fine-tuned \
        --pure_bf16 \
        --use_fast_kernels

which is a small modification of the example code in the docs here.

Within the stack trace is

File "/usr/local/lib/python3.10/dist-packages/llama_recipes/finetuning.py", line 109, in main
    model = BetterTransformer.transform(model) model = BetterTransformer.transform(model)

which seems unexpected since this is not the code I see in the source.

I installed by doing

python3 -m pip install --upgrade pip
pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes

[HamidShojanazeri]

@devon-research sorry the code has been updated and if you install from src, pip install -e . it should be in effect.

[wukaixingxp]

Hi! It seems that a solution has been provided to the issue and there has not been a follow-up conversation for a long time. I will close this issue for now and feel free to reopen it if you have any questions!