LLaVA
LLaVA copied to clipboard
haotian-liu
Error when install flash-attn
When I run pip intall flash-attn, it raises an error:
ERROR: Could not build wheels for flash-attn, which is required to install pyproject.toml-based projects
However, I have run pip install -e . and successfully installed llava. Do you know how to solve this problem?
Hi @Richar-Du, thank you for your interest in our work.
flash-attn is not required for running the inference of LLaVA, so that error message you were seeing regarding flash-attn should be not related to LLaVA at all (including the pyproject.toml part).
Can you provide: (1) the full error log, and wrap with ``` as well; (2) your system environment, including the OS, CUDA version, and GPU type?
The following is my full error log
(llava) E:\LLaVA>pip install flash-attn Collecting flash-attn Using cached flash_attn-1.0.3.post0.tar.gz (2.0 MB) Preparing metadata (setup.py) ... done Requirement already satisfied: torch in c:\users\admin.conda\envs\llava\lib\site-packages (from flash-attn) (2.0.0) Collecting einops (from flash-attn) Using cached einops-0.6.1-py3-none-any.whl (42 kB) Requirement already satisfied: packaging in c:\users\admin.conda\envs\llava\lib\site-packages (from flash-attn) (23.1) Requirement already satisfied: filelock in c:\users\admin.conda\envs\llava\lib\site-packages (from torch->flash-attn) (3.12.0) Requirement already satisfied: typing-extensions in c:\users\admin.conda\envs\llava\lib\site-packages (from torch->flash-attn) (4.5.0) Requirement already satisfied: sympy in c:\users\admin.conda\envs\llava\lib\site-packages (from torch->flash-attn) (1.11.1) Requirement already satisfied: networkx in c:\users\admin.conda\envs\llava\lib\site-packages (from torch->flash-attn) (3.1) Requirement already satisfied: jinja2 in c:\users\admin.conda\envs\llava\lib\site-packages (from torch->flash-attn) (3.1.2) Requirement already satisfied: MarkupSafe>=2.0 in c:\users\admin.conda\envs\llava\lib\site-packages (from jinja2->torch->flash-attn) (2.1.2) Requirement already satisfied: mpmath>=0.19 in c:\users\admin.conda\envs\llava\lib\site-packages (from sympy->torch->flash-attn) (1.3.0) Building wheels for collected packages: flash-attn Building wheel for flash-attn (setup.py) ... error error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [127 lines of output] No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1'
Warning: Torch did not find available GPUs on this system.
If your intention is to cross-compile, this is not an error.
By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),
Volta (compute capability 7.0), Turing (compute capability 7.5),
and, if the CUDA version is >= 11.0, Ampere (compute capability 8.0).
If you wish to cross-compile for a single specific architecture,
export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.
torch.__version__ = 2.0.0+cpu
fatal: not a git repository (or any of the parent directories): .git
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-310
creating build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\attention_kernl.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\bert_padding.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attention.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_interface.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton_og.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton_single_query.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton_tmp.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton_tmp_og.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_attn_triton_varlen.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_blocksparse_attention.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\flash_blocksparse_attn_interface.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\fused_softmax.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\rotary.py -> build\lib.win-amd64-cpython-310\flash_attn
copying flash_attn\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn
creating build\lib.win-amd64-cpython-310\flash_attn\layers
copying flash_attn\layers\patch_embed.py -> build\lib.win-amd64-cpython-310\flash_attn\layers
copying flash_attn\layers\rotary.py -> build\lib.win-amd64-cpython-310\flash_attn\layers
copying flash_attn\layers\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\layers
creating build\lib.win-amd64-cpython-310\flash_attn\losses
copying flash_attn\losses\cross_entropy.py -> build\lib.win-amd64-cpython-310\flash_attn\losses
copying flash_attn\losses\cross_entropy_apex.py -> build\lib.win-amd64-cpython-310\flash_attn\losses
copying flash_attn\losses\cross_entropy_parallel.py -> build\lib.win-amd64-cpython-310\flash_attn\losses
copying flash_attn\losses\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\losses
creating build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\bert.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\gpt.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\gptj.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\gpt_j.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\gpt_neox.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\llama.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\opt.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\vit.py -> build\lib.win-amd64-cpython-310\flash_attn\models
copying flash_attn\models\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\models
creating build\lib.win-amd64-cpython-310\flash_attn\modules
copying flash_attn\modules\block.py -> build\lib.win-amd64-cpython-310\flash_attn\modules
copying flash_attn\modules\embedding.py -> build\lib.win-amd64-cpython-310\flash_attn\modules
copying flash_attn\modules\mha.py -> build\lib.win-amd64-cpython-310\flash_attn\modules
copying flash_attn\modules\mlp.py -> build\lib.win-amd64-cpython-310\flash_attn\modules
copying flash_attn\modules\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\modules
creating build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\activations.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\fused_dense.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\gelu_activation.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\layer_norm.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\rms_norm.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
copying flash_attn\ops\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\ops
creating build\lib.win-amd64-cpython-310\flash_attn\triton
copying flash_attn\triton\fused_attention.py -> build\lib.win-amd64-cpython-310\flash_attn\triton
copying flash_attn\triton\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\triton
creating build\lib.win-amd64-cpython-310\flash_attn\utils
copying flash_attn\utils\benchmark.py -> build\lib.win-amd64-cpython-310\flash_attn\utils
copying flash_attn\utils\distributed.py -> build\lib.win-amd64-cpython-310\flash_attn\utils
copying flash_attn\utils\generation.py -> build\lib.win-amd64-cpython-310\flash_attn\utils
copying flash_attn\utils\pretrained.py -> build\lib.win-amd64-cpython-310\flash_attn\utils
copying flash_attn\utils\__init__.py -> build\lib.win-amd64-cpython-310\flash_attn\utils
running build_ext
C:\Users\admin\.conda\envs\llava\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] 系统找不到指定的文件。
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\admin\AppData\Local\Temp\pip-install-tp31ysl7\flash-attn_b02fe69769be4953b2cf3debf59648b6\setup.py", line 163, in <module>
setup(
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
return run_commands(dist)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
dist.run_commands()
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
self.run_command(cmd)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\dist.py", line 1208, in run_command
super().run_command(command)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\wheel\bdist_wheel.py", line 325, in run
self.run_command("build")
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\dist.py", line 1208, in run_command
super().run_command(command)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\command\build.py", line 132, in run
self.run_command(cmd_name)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\dist.py", line 1208, in run_command
super().run_command(command)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\command\build_ext.py", line 84, in run
_build_ext.run(self)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
self.build_extensions()
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\torch\utils\cpp_extension.py", line 383, in _check_cuda_version
torch_cuda_version = packaging.version.parse(torch.version.cuda)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 49, in parse
return Version(version)
File "C:\Users\admin\.conda\envs\llava\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 264, in __init__
match = self._regex.search(version)
TypeError: expected string or bytes-like object
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for flash-attn Running setup.py clean for flash-attn Failed to build flash-attn ERROR: Could not build wheels for flash-attn, which is required to install pyproject.toml-based projects
Hi it seems that it's a Windows machine, and it cannot find the CUDA/GPU? I am not familiar with compiling these on Windows so I may not be able to offer much help on this.
One thing I would like to mention is that the flash attention is only needed for training. So you may go ahead without installing the flash-attn and run the demo/inference.
Hi @Richar-Du, thank you for your interest in our work.
flash-attnis not required for running the inference of LLaVA, so that error message you were seeing regarding flash-attn should be not related to LLaVA at all (including thepyproject.tomlpart).Can you provide: (1) the full error log, and wrap with ``` as well; (2) your system environment, including the OS, CUDA version, and GPU type?
I want to run the training code and the error is: flash-attn.log
The OS is CentOS Linux release 7.6.1810 (Core) x86_64, the CUDA is 11.4, and the GPU is NVIDIA A100-SXM4-80GB.
Thanks in advance.
Hi @Richar-Du, sorry I just saw your comment. I am not sure if this is the cause, as it is an issue with the flash-attn and not our repo itself. But it seems that you may need a newer GCC compiler. Can you try use a newer gcc compiler in $PATH and rerun pip install? Thanks!
/home/hadoop-ba-dealrank/dolphinfs_hdd_hadoop-ba-dealrank/duyifan04/miniconda3/envs/llava/lib/python3.8/site-packages/torch/include/c10/util/C++17.h:16:2: error: #error "You're trying to build PyTorch with a too old version of GCC. We need GCC 5 or later."
#error \
^
@Richar-Du Maybe you can checkout your NVCC version, you'd better use NVCC>11.7, hope to help you!
I meet the same problem. Here's how to solve it. 1、You should install cuda11.7, cuda12 or above will failed. 2、You should install gcc11 and g++11 or below, as cuda11.7 required. You should run g++ --version to make sure you install the correct version. 3、conda install -c nvidia/label/cuda-11.7.0 cuda-nvcc 4、export CPATH=/usr/local/cuda-11.7/targets/x86_64-linux/include:$CPATH export LD_LIBRARY_PATH=/usr/local/cuda-11.7/targets/x86_64-linux/lib:$LD_LIBRARY_PATH export PATH=/usr/local/cuda-11.7/bin:$PATH 5、Finally, run pip install flash-attn
