unilm icon indicating copy to clipboard operation
unilm copied to clipboard
verified publisher icon microsoft

[Kosmos-v2] unable to build the environment

Open BrainWWW opened this issue 2 years ago • 7 comments

I build the environment using the docker method. But when it goes to pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers, I get the below error:

building 'xformers._C_flashattention' extension
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn
  creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src
  Emitting ninja build file /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/build.ninja...
  Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/33] /usr/local/cuda/bin/nvcc  -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/cutlass/include -I/opt/conda/lib/python3.8/site-packages/torch/include -I/opt/conda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.8 -c -c /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_hdim160_bf16_sm80.cu -o /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_hdim160_bf16_sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DHAS_PYTORCH --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --extended-lambda -D_ENABLE_EXTENDED_ALIGNED_STORAGE --generate-line-info -DNDEBUG --threads 4 --ptxas-options=-v -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 --generate-line-info -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C_flashattention -D_GLIBCXX_USE_CXX11_ABI=0
  FAILED: /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_hdim160_bf16_sm80.o
  /usr/local/cuda/bin/nvcc  -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src -I/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/cutlass/include -I/opt/conda/lib/python3.8/site-packages/torch/include -I/opt/conda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.8 -c -c /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_hdim160_bf16_sm80.cu -o /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8/tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_hdim160_bf16_sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DHAS_PYTORCH --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --extended-lambda -D_ENABLE_EXTENDED_ALIGNED_STORAGE --generate-line-info -DNDEBUG --threads 4 --ptxas-options=-v -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 --generate-line-info -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C_flashattention -D_GLIBCXX_USE_CXX11_ABI=0
  /opt/conda/lib/python3.8/site-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
            detected during:
              instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, <unnamed>>::operator==(const c10::detail::integer_iterator<I, one_sided, <unnamed>> &) const [with I=size_t, one_sided=false, <unnamed>=0]"
  (61): here
              instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, <unnamed>>::operator!=(const c10::detail::integer_iterator<I, one_sided, <unnamed>> &) const [with I=size_t, one_sided=false, <unnamed>=0]"
  /opt/conda/lib/python3.8/site-packages/torch/include/c10/core/TensorImpl.h(77): here

  /opt/conda/lib/python3.8/site-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
            detected during:
              instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, <unnamed>>::operator==(const c10::detail::integer_iterator<I, one_sided, <unnamed>> &) const [with I=size_t, one_sided=true, <unnamed>=0]"
  (61): here
              instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, <unnamed>>::operator!=(const c10::detail::integer_iterator<I, one_sided, <unnamed>> &) const [with I=size_t, one_sided=true, <unnamed>=0]"
  /opt/conda/lib/python3.8/site-packages/torch/include/c10/core/TensorImpl.h(2327): here
              instantiation of "__nv_bool c10::TensorImpl::SetDimsTemplate(c10::ArrayRef<T>) [with T=int64_t, <unnamed>=void]"
  /opt/conda/lib/python3.8/site-packages/torch/include/c10/core/TensorImpl.h(2337): here

It seems that the issue is caused by a version mismatch between CUDA and xformers. I don't know how to solve it. Please help me.

BrainWWW avatar Aug 28 '23 08:08 BrainWWW

Hi, @BrainWWW. Can https://github.com/microsoft/unilm/issues/1253#issuecomment-1679956365 help?

pengzhiliang avatar Aug 28 '23 08:08 pengzhiliang

yep, I tried it but got the same error.

BrainWWW avatar Aug 28 '23 08:08 BrainWWW

Hi, @BrainWWW. I run the following code to install xformers:

pip install -v -U git+https://github.com/facebookresearch/xformers.git@82254f4b0d9c625f7efa8d6671f58144e441901d#egg=xformers

And It works: image

pengzhiliang avatar Aug 28 '23 08:08 pengzhiliang

hi, I used the same commands to install xformers but it didn't work. @pengzhiliang

Is the flash_attention package pre-installed in your image? The issue was caused by the installation of flash_attention package.

BrainWWW avatar Aug 28 '23 09:08 BrainWWW

@BrainWWW, the image is just from nvcr.io/nvidia/pytorch:22.10-py3.

I notice that your error info is from xformer 896a4241413344a4850e6654ebe11206 commit.

building 'xformers._C_flashattention' extension creating /tmp/pip-install-lwdzzryz/xformers_896a4241413344a4850e6654ebe11206/build/temp.linux-x86_64-3.8

But the commit I used is 82254f4b0d9c625f7efa8d6671f58144e441901d.

pengzhiliang avatar Aug 28 '23 10:08 pengzhiliang

Some errors are raised when flashattention v2 was merged in xformer. So I use a previous one.

pengzhiliang avatar Aug 28 '23 10:08 pengzhiliang

Hi @pengzhiliang,

I am trying to install kosmos-2 using the docker setup instructions. However, after downloading the mentioned image and running bash vl_setup_xl.sh , xformers does not get installed. I tried to install xformers from the commit you mentioned here, however this also does not work. Kindly let me know if you have any suggestions.


RuntimeError: Error compiling objects for extension
    Running setup.py install for xformers ... error
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0ldrt5p8/xformers_60d1f7e3288344aa8895d4b26be4916d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0ldrt5p8/xformers_60d1f7e3288344aa8895d4b26be4916d/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mzfxv4zy/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.8/xformers Check the logs for full command output.

Hi, @BrainWWW. I run the following code to install xformers:

pip install -v -U git+https://github.com/facebookresearch/xformers.git@82254f4b0d9c625f7efa8d6671f58144e441901d#egg=xformers

And It works: image

PoulamiSM avatar Jul 01 '24 13:07 PoulamiSM