Aman Karmani comments

Results 460 comments of


                                            Aman Karmani

add flash attention

ah i think you need to call `replace...` method to monkey patch *before* the model is instantiated, i.e. before `AutoModelForCausalLM.from_pretrained`

add flash attention

try this? https://github.com/artidoro/qlora/commit/1b5641913914a48ad15eadb96a0dba6452aa0ac1

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

Try this: > pip install -U wheel

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

> Try this: > > > pip install -U wheel someone reported this worked in their environment. but when we tried in a fresh docker/conda env, its not working. nor...

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

Use: > pip install -U flash-attn --no-build-isolation

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

The issue here is that once you add a pyproject.toml, pip will use that and use build isolation. To make isolation work, we would need to add to the toml:...

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

fixed by https://github.com/Dao-AILab/flash-attention/commit/73bd3f3bbb6775c5286e4b095efbc62d9fd4e5dd

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

The fix is out. Try: `pip install -v flash-attn==2.1.1`

pip install flash-attn always happens ModuleNotFoundError: No module named 'packaging',but actually i have pip install packaging

@tridao this issue can be closed. If you want to give me issue maint privileges, I can help out keeping things tidy.

I thought nvcc wasnt required @ 2.0.7 but is again?

Weird, can you run with `-v` and post the output