sam = sam_model_registry[model_type](checkpoint=sam_checkpoint) is giving me this error

[fran-zeitview]

Error(s) in loading state_dict for Sam: Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

[mattyamonaca]

I'm having the same problem. File "/xxx/xxx/xxxx/xxxxx/xxxx/xxxx/xxxxx.py", line 22, in get_mask_generator sam = sam_model_registrymodel_type File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/segment_anything/build_sam.py", line 15, in build_sam_vit_h return _build_sam( File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/segment_anything/build_sam.py", line 106, in _build_sam sam.load_state_dict(state_dict) File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Sam: Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

[thangngoc89]

This is because you have the original sam installed. Try to remove facebook/segment-anything before installing this or just try in a fresh python env (conda or pipenv)

[ymq2017]

Hi, it seems that the segment anything you load is not in sam-hq. You can use the following line of command and run it again.

export PYTHONPATH=$(pwd)

[richard-schwab]

I'm getting the same and I made sure to uninstall the segment-anything:

/home/user/.pyenv/versions/3.10.4/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
Traceback (most recent call last):
  File "/nns/img_seg/sam-hq/demo/demo_sam.py", line 63, in <module>
    sam = sam_model_registry_baseline[model_type](checkpoint=sam_checkpoint)
  File "/nns/img_seg/sam-hq/segment_anything/build_sam_baseline.py", line 28, in build_sam_vit_l
    return _build_sam(
  File "/nns/img_seg/sam-hq/segment_anything/build_sam_baseline.py", line 106, in _build_sam
    sam.load_state_dict(state_dict)
  File "/home/user/.pyenv/versions/3.10.4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Sam:
	Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

As for the error with libtorch_cuda_cu.so, perhaps you can add a requirements.txt so we know which versions to install?

[ymq2017]

I'm getting the same and I made sure to uninstall the segment-anything:

/home/user/.pyenv/versions/3.10.4/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
Traceback (most recent call last):
  File "/nns/img_seg/sam-hq/demo/demo_sam.py", line 63, in <module>
    sam = sam_model_registry_baseline[model_type](checkpoint=sam_checkpoint)
  File "/nns/img_seg/sam-hq/segment_anything/build_sam_baseline.py", line 28, in build_sam_vit_l
    return _build_sam(
  File "/nns/img_seg/sam-hq/segment_anything/build_sam_baseline.py", line 106, in _build_sam
    sam.load_state_dict(state_dict)
  File "/home/user/.pyenv/versions/3.10.4/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Sam:
	Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

As for the error with libtorch_cuda_cu.so, perhaps you can add a requirements.txt so we know which versions to install?

Hi, demo_sam.py is the baseline result for the original SAM. To run this script, you need to download baseline SAM checkpoint as described at the end of https://github.com/SysCV/sam-hq#getting-started . If you want to load our hq-sam checkpoint, you could use demo_hqsam.py . We also provide a more detailed colab notebook now.

[richard-schwab]

Hi, demo_sam.py is the baseline result for the original SAM. To run this script, you need to download baseline SAM checkpoint as described at the end of https://github.com/SysCV/sam-hq#getting-started . If you want to load our hq-sam checkpoint, you could use demo_hqsam.py . We also provide a more detailed colab notebook now.

You were correct this fixed the problem. I hadn't downloaded the original as well. Thank you for that. Here is the link for that model in case anyone else needs: https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth

[nykgaurav]

I am also facing the same issue. Are there permanent solution to this problem? I tried uninstalling segment-anything before installing this sam-hq, but getting the same error

[a373648449]

KeyError: 'groundingdino_swint' Start SAM Processing Found GroundingDINO in pip. Verifying if dynamic library build success. GroundingDINO dynamic library have been successfully built. Running GroundingDINO Inference Initializing GroundingDINO GroundingDINO_SwinB (938MB) final text_encoder_type: bert-base-uncased C:\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py:884: FutureWarning: The device argument i s deprecated and will be removed in v5 of Transformers. warnings.warn( Initializing SAM to cuda {'sam_vit_h': <function build_sam_vit_h at 0x000001F918FD4160>, 'sam_vit_l': <function build_sam_vit_l at 0x000001F918FD41F 0>, 'sam_vit_b': <function build_sam_vit_b at 0x000001F918FA76D0>, 'sam_hq_vit_h': <function build_sam_hq_vit_h at 0x000001 F918E8D120>, 'sam_hq_vit_l': <function build_sam_hq_vit_l at 0x000001F918FEDFC0>, 'sam_hq_vit_b': <function build_sam_hq_vi t_b at 0x000001F918FEF2E0>, 'mobile_sam': <function build_mobile_sam at 0x000001F918FEF370>} Traceback (most recent call last): File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api result = await self.call_function( File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function prediction = await anyio.to_thread.run_sync( File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thre ad return await future File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, *args) File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper response = f(*args, **kwargs) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 207, in sam_predict sam = init_sam_model(sam_model_name) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 130, in init_sam_model
sam_model_cache[sam_model_name] = load_sam_model(sam_model_name) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 81, in load_sam_model sam = sam_model_registrymodel_type KeyError: 'groundingdino_swint'

I also encountered this exception and it was solved

[mzg0108]

KeyError: 'groundingdino_swint' Start SAM Processing Found GroundingDINO in pip. Verifying if dynamic library build success. GroundingDINO dynamic library have been successfully built. Running GroundingDINO Inference Initializing GroundingDINO GroundingDINO_SwinB (938MB) final text_encoder_type: bert-base-uncased C:\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py:884: FutureWarning: The device argument i s deprecated and will be removed in v5 of Transformers. warnings.warn( Initializing SAM to cuda {'sam_vit_h': <function build_sam_vit_h at 0x000001F918FD4160>, 'sam_vit_l': <function build_sam_vit_l at 0x000001F918FD41F 0>, 'sam_vit_b': <function build_sam_vit_b at 0x000001F918FA76D0>, 'sam_hq_vit_h': <function build_sam_hq_vit_h at 0x000001 F918E8D120>, 'sam_hq_vit_l': <function build_sam_hq_vit_l at 0x000001F918FEDFC0>, 'sam_hq_vit_b': <function build_sam_hq_vi t_b at 0x000001F918FEF2E0>, 'mobile_sam': <function build_mobile_sam at 0x000001F918FEF370>} Traceback (most recent call last): File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api result = await self.call_function( File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function prediction = await anyio.to_thread.run_sync( File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thre ad return await future File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, *args) File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper response = f(*args, **kwargs) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 207, in sam_predict sam = init_sam_model(sam_model_name) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 130, in init_sam_model sam_model_cache[sam_model_name] = load_sam_model(sam_model_name) File "C:\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\sam.py", line 81, in load_sam_model sam = sam_model_registrymodel_type KeyError: 'groundingdino_swint'

I also encountered this exception and it was solved

How was it solved?

[mzg0108]

I am also facing the same issue. Are there permanent solution to this problem? I tried uninstalling segment-anything before installing this sam-hq, but getting the same error

Have you found any solution?

[mzg0108]

I'm having the same problem. File "/xxx/xxx/xxxx/xxxxx/xxxx/xxxx/xxxxx.py", line 22, in get_mask_generator sam = sam_model_registrymodel_type File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/segment_anything/build_sam.py", line 15, in build_sam_vit_h return _build_sam( File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/segment_anything/build_sam.py", line 106, in _build_sam sam.load_state_dict(state_dict) File "/xxxx/xxxx/.pyenv/versions/3.10.8/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Sam: Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

Did you find the solution?

[mzg0108]

I have added strict=False to the load_state_dict function call and apparently it is working. I guess the error was because of different versions of pytorch.

[squashking]

Any solutions so far?

I have encountered the same error:

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Sam: Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

[squashking]

I had the same issue. In my case, my code used original SAM and then I wanted to change for HQ-SAM, but I didn't update the import. So the fix was just to change:

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor into: from segment_anything_hq import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor