xformers icon indicating copy to clipboard operation
xformers copied to clipboard

Published 20 hours ago • verified publisher icon facebookresearch

Is this a mistake in requirement of llama_inference?

Open pengjiao123 opened this issue 6 months ago • 2 comments

https://github.com/facebookresearch/xformers/blob/main/examples/llama_inference/requirements.txt

torch>=2.2.0 this should be a mistake
we should use FlashAttention >=2.2.0

pengjiao123 avatar Dec 11 '23 11:12 pengjiao123

Hi, I believe this is on-purpose. There is a bug in PyTorch 2.1 which prevents CUDA GRaphs from working properly with NCCL collectives. It will be fixed in 2.2, but in the meantime you can use the nightly (which are already tagged as 2.2)

danthe3rd avatar Dec 11 '23 12:12 danthe3rd

Hi, I believe this is on-purpose. There is a bug in PyTorch 2.1 which prevents CUDA GRaphs from working properly with NCCL collectives. It will be fixed in 2.2, but in the meantime you can use the nightly (which are already tagged as 2.2)

ok,thanks

pengjiao123 avatar Dec 12 '23 06:12 pengjiao123