Grounded-Segment-Anything
Grounded-Segment-Anything copied to clipboard
IDEA-Research
centos7 Failed to load custom C++ ops. Running on CPU mode Only
question
i have a super GPU machine to run model, however, when i install all pkgs by following the commands in the repo's readme.md, i met an eccentric problem, i can't use my gpu! can you help me solve the problem, so that i can run the project on my GPU ?
env
python 3.10
torch 2.0.0
centos7
gcc version 4.8.5
error
/home/lcc/github.com/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
/home/lcc/miniconda3/envs/gsa/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1678402312629/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
File "/home/lcc/github.com/Grounded-Segment-Anything/grounding_dino_demo.py", line 152, in <module>
model = load_model(config_file, grounded_checkpoint, device=device)
File "/home/lcc/github.com/Grounded-Segment-Anything/grounding_dino_demo.py", line 75, in load_model
checkpoint = torch.load(model_checkpoint_path, map_location="cpu")
File "/home/lcc/miniconda3/envs/gsa/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/lcc/miniconda3/envs/gsa/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x1f'.
it seems like some python package's error, maybe you can check the version of these packages, here is my env info:
Package Version
----------------------- ------------------
absl-py 1.4.0
addict 2.4.0
aiofiles 23.1.0
aiohttp 3.8.4
aiosignal 1.3.1
altair 4.2.2
antlr4-python3-runtime 4.9.3
anyio 3.6.2
asttokens 2.0.8
async-timeout 4.0.2
attrs 22.2.0
backcall 0.2.0
black 23.3.0
brotlipy 0.7.0
cachetools 5.3.0
certifi 2022.12.7
cffi 1.15.1
charset-normalizer 2.0.4
click 8.1.3
cloudpickle 2.2.1
cmake 3.26.3
coloredlogs 15.0.1
contourpy 1.0.7
controlnet-aux 0.0.2
cryptography 39.0.1
cycler 0.11.0
dataclasses-json 0.5.7
debugpy 1.6.3
decorator 5.1.1
diffusers 0.14.0
distlib 0.3.6
einops 0.6.0
entrypoints 0.4
executing 1.1.1
fastapi 0.95.0
ffmpeg-python 0.2.0
ffmpy 0.3.0
filelock 3.10.7
flatbuffers 23.3.3
flit_core 3.8.0
fonttools 4.39.3
frozenlist 1.3.3
fsspec 2023.4.0
future 0.18.3
fvcore 0.1.5.post20221221
google-auth 2.17.2
google-auth-oauthlib 1.0.0
gradio 3.25.0
gradio_client 0.1.0
greenlet 2.0.2
grpcio 1.53.0
h11 0.14.0
httpcore 0.17.0
httpx 0.24.0
huggingface-hub 0.13.3
humanfriendly 10.0
hydra-core 1.3.2
idna 3.4
importlib-metadata 6.1.0
importlib-resources 5.12.0
iopath 0.1.9
ipykernel 6.16.0
ipython 8.5.0
jedi 0.18.1
Jinja2 3.1.2
jsonschema 4.17.3
jupyter_client 7.4.3
jupyter_core 4.11.2
kiwisolver 1.4.4
langchain 0.0.101
linkify-it-py 2.0.0
lit 16.0.1
llvmlite 0.39.1
Markdown 3.4.3
markdown-it-py 2.2.0
MarkupSafe 2.1.2
marshmallow 3.19.0
marshmallow-enum 1.5.1
matplotlib 3.7.1
matplotlib-inline 0.1.6
mdit-py-plugins 0.3.3
mdurl 0.1.2
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
more-itertools 9.1.0
mpmath 1.3.0
multidict 6.0.4
mypy-extensions 1.0.0
nest-asyncio 1.5.6
numba 0.56.4
numpy 1.23.5
oauthlib 3.2.2
omegaconf 2.3.0
onnx 1.13.1
onnxruntime 1.14.1
openai 0.27.4
openai-whisper 20230314
opencv-python 4.7.0.72
orjson 3.8.10
packaging 23.0
pandas 2.0.0
parso 0.8.3
pathspec 0.11.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.4.0
pip 23.0.1
pkgutil_resolve_name 1.3.10
platformdirs 3.2.0
portalocker 2.7.0
prompt-toolkit 3.0.31
protobuf 3.20.3
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycocotools 2.0.6
pycparser 2.21
pydantic 1.10.7
pydub 0.25.1
Pygments 2.13.0
pyOpenSSL 23.0.0
pyparsing 3.0.9
pyrsistent 0.19.3
PySocks 1.7.1
python-dateutil 2.8.2
python-multipart 0.0.6
pytz 2023.3
PyYAML 6.0
pyzmq 24.0.1
regex 2023.3.23
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
scipy 1.10.1
semantic-version 2.10.0
setuptools 65.6.3
six 1.16.0
sniffio 1.3.0
SQLAlchemy 1.4.47
stack-data 0.5.1
starlette 0.26.1
supervision 0.3.2
sympy 1.11.1
tabulate 0.9.0
tenacity 8.2.2
tensorboard 2.12.1
tensorboard-data-server 0.7.0
tensorboard-plugin-wit 1.8.1
termcolor 2.2.0
tiktoken 0.3.1
timm 0.6.13
tokenizers 0.13.3
tomli 2.0.1
toolz 0.12.0
torch 1.12.1
torchaudio 0.12.1
torchvision 0.13.1
tornado 6.2
tqdm 4.65.0
traitlets 5.5.0
transformers 4.27.4
triton 2.0.0
typing_extensions 4.4.0
typing-inspect 0.8.0
tzdata 2023.3
uc-micro-py 1.0.1
urllib3 1.26.15
uvicorn 0.21.1
virtualenv 20.21.0
wcwidth 0.2.5
websockets 11.0.1
Werkzeug 2.2.3
wheel 0.38.4
yacs 0.1.8
yapf 0.32.0
yarl 1.8.2
zipp 3.15.0
/home/lcc/github.com/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
I am assuming you ran the setup.py file, for grounding dino it builds a custom cuda extension for GPU , can tell the cuda version in your system, and the one supported by your pytorch ?
/home/lcc/github.com/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only! warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")I am assuming you ran the setup.py file, for grounding dino it builds a custom cuda extension for GPU , can tell the cuda version in your system, and the one supported by your pytorch ?
sory, i forgot to list my env and my option, now i have updated all info in the question desc, pls read it again.
and my cuda vesion is 11.4, pytorch is 2.0,
i can't run the first demo, pls help me about the question, thanks
it seems like some python package's error, maybe you can check the version of these packages, here is my env info:
Package Version ----------------------- ------------------ absl-py 1.4.0 addict 2.4.0 aiofiles 23.1.0 aiohttp 3.8.4 aiosignal 1.3.1 altair 4.2.2 antlr4-python3-runtime 4.9.3 anyio 3.6.2 asttokens 2.0.8 async-timeout 4.0.2 attrs 22.2.0 backcall 0.2.0 black 23.3.0 brotlipy 0.7.0 cachetools 5.3.0 certifi 2022.12.7 cffi 1.15.1 charset-normalizer 2.0.4 click 8.1.3 cloudpickle 2.2.1 cmake 3.26.3 coloredlogs 15.0.1 contourpy 1.0.7 controlnet-aux 0.0.2 cryptography 39.0.1 cycler 0.11.0 dataclasses-json 0.5.7 debugpy 1.6.3 decorator 5.1.1 diffusers 0.14.0 distlib 0.3.6 einops 0.6.0 entrypoints 0.4 executing 1.1.1 fastapi 0.95.0 ffmpeg-python 0.2.0 ffmpy 0.3.0 filelock 3.10.7 flatbuffers 23.3.3 flit_core 3.8.0 fonttools 4.39.3 frozenlist 1.3.3 fsspec 2023.4.0 future 0.18.3 fvcore 0.1.5.post20221221 google-auth 2.17.2 google-auth-oauthlib 1.0.0 gradio 3.25.0 gradio_client 0.1.0 greenlet 2.0.2 grpcio 1.53.0 h11 0.14.0 httpcore 0.17.0 httpx 0.24.0 huggingface-hub 0.13.3 humanfriendly 10.0 hydra-core 1.3.2 idna 3.4 importlib-metadata 6.1.0 importlib-resources 5.12.0 iopath 0.1.9 ipykernel 6.16.0 ipython 8.5.0 jedi 0.18.1 Jinja2 3.1.2 jsonschema 4.17.3 jupyter_client 7.4.3 jupyter_core 4.11.2 kiwisolver 1.4.4 langchain 0.0.101 linkify-it-py 2.0.0 lit 16.0.1 llvmlite 0.39.1 Markdown 3.4.3 markdown-it-py 2.2.0 MarkupSafe 2.1.2 marshmallow 3.19.0 marshmallow-enum 1.5.1 matplotlib 3.7.1 matplotlib-inline 0.1.6 mdit-py-plugins 0.3.3 mdurl 0.1.2 mkl-fft 1.3.1 mkl-random 1.2.2 mkl-service 2.4.0 more-itertools 9.1.0 mpmath 1.3.0 multidict 6.0.4 mypy-extensions 1.0.0 nest-asyncio 1.5.6 numba 0.56.4 numpy 1.23.5 oauthlib 3.2.2 omegaconf 2.3.0 onnx 1.13.1 onnxruntime 1.14.1 openai 0.27.4 openai-whisper 20230314 opencv-python 4.7.0.72 orjson 3.8.10 packaging 23.0 pandas 2.0.0 parso 0.8.3 pathspec 0.11.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.4.0 pip 23.0.1 pkgutil_resolve_name 1.3.10 platformdirs 3.2.0 portalocker 2.7.0 prompt-toolkit 3.0.31 protobuf 3.20.3 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycocotools 2.0.6 pycparser 2.21 pydantic 1.10.7 pydub 0.25.1 Pygments 2.13.0 pyOpenSSL 23.0.0 pyparsing 3.0.9 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-multipart 0.0.6 pytz 2023.3 PyYAML 6.0 pyzmq 24.0.1 regex 2023.3.23 requests 2.28.1 requests-oauthlib 1.3.1 rsa 4.9 scipy 1.10.1 semantic-version 2.10.0 setuptools 65.6.3 six 1.16.0 sniffio 1.3.0 SQLAlchemy 1.4.47 stack-data 0.5.1 starlette 0.26.1 supervision 0.3.2 sympy 1.11.1 tabulate 0.9.0 tenacity 8.2.2 tensorboard 2.12.1 tensorboard-data-server 0.7.0 tensorboard-plugin-wit 1.8.1 termcolor 2.2.0 tiktoken 0.3.1 timm 0.6.13 tokenizers 0.13.3 tomli 2.0.1 toolz 0.12.0 torch 1.12.1 torchaudio 0.12.1 torchvision 0.13.1 tornado 6.2 tqdm 4.65.0 traitlets 5.5.0 transformers 4.27.4 triton 2.0.0 typing_extensions 4.4.0 typing-inspect 0.8.0 tzdata 2023.3 uc-micro-py 1.0.1 urllib3 1.26.15 uvicorn 0.21.1 virtualenv 20.21.0 wcwidth 0.2.5 websockets 11.0.1 Werkzeug 2.2.3 wheel 0.38.4 yacs 0.1.8 yapf 0.32.0 yarl 1.8.2 zipp 3.15.0
i use conda as my pkg manage tool. i use command pip list to show my pkg list:
accelerate 0.18.0
addict 2.4.0
asttokens 2.2.1
backcall 0.2.0
brotlipy 0.7.0
certifi 2022.12.7
cffi 1.15.1
charset-normalizer 2.0.4
coloredlogs 15.0.1
comm 0.1.3
contourpy 1.0.7
cryptography 39.0.1
cycler 0.11.0
debugpy 1.6.7
decorator 5.1.1
diffusers 0.15.1
executing 1.2.0
filelock 3.9.0
flatbuffers 23.3.3
flit_core 3.8.0
fonttools 4.39.3
gmpy2 2.1.2
groundingdino 0.1.0 /home/lcc/github.com/GroundingDINO
huggingface-hub 0.13.4
humanfriendly 10.0
idna 3.4
importlib-metadata 6.5.0
ipykernel 6.22.0
ipython 8.12.0
jedi 0.18.2
Jinja2 3.1.2
jupyter_client 8.2.0
jupyter_core 5.3.0
kiwisolver 1.4.4
MarkupSafe 2.1.1
matplotlib 3.7.1
matplotlib-inline 0.1.6
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mpmath 1.2.1
nest-asyncio 1.5.6
networkx 2.8.4
numpy 1.23.5
onnx 1.13.1
onnxruntime 1.14.1
opencv-python 4.7.0.72
packaging 23.1
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.4.0
pip 23.0.1
platformdirs 3.2.0
prompt-toolkit 3.0.38
protobuf 3.20.3
psutil 5.9.5
ptyprocess 0.7.0
pure-eval 0.2.2
pycocotools 2.0.6
pycparser 2.21
Pygments 2.15.1
pyOpenSSL 23.0.0
pyparsing 3.0.9
PySocks 1.7.1
python-dateutil 2.8.2
PyYAML 6.0
pyzmq 25.0.2
regex 2023.3.23
requests 2.28.1
segment-anything 1.0 /home/lcc/github.com/segment-anything
setuptools 66.0.0
six 1.16.0
stack-data 0.6.2
supervision 0.4.0
sympy 1.11.1
timm 0.6.13
tokenizers 0.13.3
tomli 2.0.1
torch 2.0.0
torchaudio 2.0.0
torchvision 0.15.0
tornado 6.3
tqdm 4.65.0
traitlets 5.9.0
transformers 4.28.1
typing_extensions 4.4.0
urllib3 1.26.15
wcwidth 0.2.6
wheel 0.38.4
yapf 0.33.0
zipp 3.15.0
so after a comparison, i find some difference between us, i list the pkg which i think it will be the key point:
# my
torch 2.0.0
diffusers 0.15.1
# your
torch 1.12.1
diffusers 0.14.0
the point is that i don't know how can i properly install all the pkg, and which pkg i had installed need to upgrade. could you give me some advice, thanks,
Try to use a virtual environment , such as pipenv , and install all the packages in this environment using pip , also if I assume you are using torch 2.0.0, it supports two cuda version 11.7 and 11.8 only , so kindly check which cuda version your pytorch supports and use that particular cuda only. Building a custom cuda extension requires a local cuda toolkit installed in the system which matches your pytorch as well
I will list down the steps I followed ,from scratch . I tried running the notebook .
Step 1-
Setup Venv in this directory by using pipenv .
Step 2
Activate environment, along with that install jupyter if using notebook
Step 3
Run the setup files as mentioned in the notebook for both grounding dino and sam.
Step 4
Verify if any _C file is created in grounding dino file , it's a custom CUDA extension that is build by grounding dino
I used pytorch nightly builds along with supporting version of CUDA 11.8 , hence I installed CUDA toolkit 11.8 as well .
Lastly setup CUDA_HOME and LD_LIBRARY paths which will be needed to local cuda toolkit and it's libraries . https://forums.developer.nvidia.com/t/path-ld-library-path/48080
In my case I wrote the following lines in the .bashrc file to set up the above two variables
export PATH=/usr/local/cuda/bin:$PATH
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
This is the path
GroundingDINO/groundingdino/_C.cpython-310-x86_64-linux-gnu.so
you should have something like this _C file which indicates CUDA extension was built successfully .
I will list down the steps I followed ,from scratch . I tried running the notebook . Step 1- Setup Venv in this directory by using
pipenv. Step 2 Activate environment, along with that install jupyter if using notebook Step 3 Run the setup files as mentioned in the notebook for both grounding dino and sam. Step 4 Verify if any_Cfile is created in grounding dino file , it's a custom CUDA extension that is build by grounding dinoI used pytorch nightly builds along with supporting version of CUDA 11.8 , hence I installed CUDA toolkit 11.8 as well .
Lastly setup CUDA_HOME and LD_LIBRARY paths which will be needed to local cuda toolkit and it's libraries . https://forums.developer.nvidia.com/t/path-ld-library-path/48080
In my case I wrote the following lines in the .bashrc file to set up the above two variables
export PATH=/usr/local/cuda/bin:$PATH export CUDA_HOME=/usr/local/cuda export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATHThis is the path
GroundingDINO/groundingdino/_C.cpython-310-x86_64-linux-gnu.soyou should have something like this _C file which indicates CUDA extension was built successfully .
first
thans your reply, i try to install pytorch matches my cuda, then i run the same demo, i get an same error except the failed to load c++ use cpu only. i can use gpu ! however, the ' invalid load key \x1f' error still exist,
how can i solve this or get more information about it?
my bashrc define
# cuda 11.7
export CUDA_HOME=/home/lcc/miniconda3/envs/py310cuda117
export PATH=/home/lcc/miniconda3/envs/py310cuda117/bin:$PATH
export LD_LIBRARY_PATH=/home/lcc/miniconda3/envs/py310cuda117/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/home/lcc/miniconda3/envs/py310cuda117/lib:$LIBRARY_PATH
my env
cuda=11.7
gcc=10.4
g++=10.4
python=3.10
pytorch=2.0
my command and error desc.
python grounding_dino_demo.py \
> --config GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py \
> --grounded_checkpoint groundingdino_swint_ogc.pth \
> --input_image assets/demo1.jpg \
> --output_dir "outputs" \
> --box_threshold 0.3 \
> --text_threshold 0.25 \
> --text_prompt "bear" \
> --device "cuda"
/home/lcc/miniconda3/envs/py310cuda117/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1678411187366/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
File "/home/lcc/github.com/Grounded-Segment-Anything/grounding_dino_demo.py", line 152, in <module>
model = load_model(config_file, grounded_checkpoint, device=device)
File "/home/lcc/github.com/Grounded-Segment-Anything/grounding_dino_demo.py", line 75, in load_model
checkpoint = torch.load(model_checkpoint_path, map_location="cpu")
File "/home/lcc/miniconda3/envs/py310cuda117/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/lcc/miniconda3/envs/py310cuda117/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x1f'.
Can you try to run the same script by giving device argument as cpu and check if it's running properly or not?
Can you try to run the same script by giving device argument as
cpuand check if it's running properly or not?
thanks, i downloaded model again, and i found the new model size is bigger than the old one. then i reinstalled python3.9+cuda11.3, all error has gone!
I have the same question when I use python 3.10... The error has gone in 3.8, I feel so confuse. Why this happen?
cd GroundingDINO export PATH=/usr/local/cuda/bin:$PATH export CUDA_HOME=/usr/local/cuda export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
pip install -e .
And then I fixed all error...... That is crazy!
For me, only the combination of the exported env and proper torch \ torchvision versions solved the problem!
export CUDA_HOME=/usr/local/cuda && export TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0;7.5;8.0;8.6+PTX;8.9;9.0" && git clone https://github.com/IDEA-Research/GroundingDINO.git && cd GroundingDINO && pip install -e .
and requirements.txt:
torch==2.0.1
torchvision==0.15.2 # <--- did not work until I set the version to be 0.15.2
