vikhyat
Error "At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`"
Hi,
This project looks cool & interesting so was trying it out - I don't know if this is an actual issue or just an issue with my configuration, but when I run it (after doing pip install -r requirements.txt) I get the following error (I get the same error both on my 8GB M2 Mac Mini Sonoma 14.1.2 and my 8GB M1 MacBook Air Monterey 12.7.1, and I get it both if I try run in a virtual environment, or just with the global Python setup):
Have I done something wrong? Is there something extra I am meant to download? Anyone else get this? How much disk space should I have?
(moon) david@Davids-Mac-mini moondream % ./python3 sample.py --image assets/demo1.jpg --interactive
Fetching 12 files: 100%|...| 12/12 [00:00<00:00, 82646.38it/s]
model_path=/Users/david/.cache/huggingface/hub/models--vikhyatk--moondream0/snapshots/c875d938535a26218a62d8a4c21818f3622dba6c
Traceback (most recent call last):
File "/Users/david/moondream/sample.py", line 12, in <module>
text_model = TextModel(model_path)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/text_model.py", line 22, in __init__
self.model = load_checkpoint_and_dispatch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/big_modeling.py", line 545, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 1373, in load_checkpoint_in_model
raise ValueError(
ValueError: At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.
More info:
(moon) david@Davids-Mac-mini moondream % python3 --version
Python 3.11.6
(moon) david@Davids-Mac-mini moondream % which python3
/Users/david/moondream/moon/bin/python3
(moon) david@Davids-Mac-mini moondream % which pip
/Users/david/moondream/moon/bin/pip
On M2 Mac I got it to run further by doing the following, adding an offload_folder but not sure if this is correct (also freed some space, not sure if that's also why):
print(f"model_path={model_path}")
self.model = load_checkpoint_and_dispatch(
self.model,
f"{model_path}/text_model.pt",
device_map="auto",
offload_folder='/Users/david/moondream/offload'
)
Now it gets as far as prompting, but now I get new error:
model_path=/Users/david/.cache/huggingface/hub/models--vikhyatk--moondream0/snapshots/c875d938535a26218a62d8a4c21818f3622dba6c
WARNING:root:Some parameters are on the meta device device because they were offloaded to the disk.
> Tell me about the image
Traceback (most recent call last):
File "/Users/david/moondream/sample.py", line 25, in <module>
print(text_model.answer_question(image_embeds, question))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/text_model.py", line 101, in answer_question
answer = self.generate(
^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/text_model.py", line 73, in generate
output_ids = self.model.generate(
^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/transformers/generation/utils.py", line 1718, in generate
return self.greedy_search(
^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/transformers/generation/utils.py", line 2579, in greedy_search
outputs = self(
^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/phi/modeling_phi.py", line 956, in forward
hidden_states = self.transformer(
^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/phi/modeling_phi.py", line 915, in forward
hidden_states = layer(
^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/phi/modeling_phi.py", line 743, in forward
feed_forward_hidden_states = self.resid_dropout(self.mlp(hidden_states))
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moondream/phi/modeling_phi.py", line 339, in forward
hidden_states = self.fc2(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/hooks.py", line 160, in new_forward
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/hooks.py", line 293, in pre_forward
set_module_tensor_to_device(
File "/Users/david/moondream/moon/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 317, in set_module_tensor_to_device
new_value = value.to(device)
^^^^^^^^^^^^^^^^
TypeError: BFloat16 is not supported on MPS
Sorry about that! Can you try changing this line to force the model to load on CPU? Looks like I need to do some more testing on MPS. https://github.com/vikhyat/moondream/blob/main/moondream/text_model.py#L24
self.model = load_checkpoint_and_dispatch(
self.model,
f"{model_path}/text_model.pt",
- device_map="auto",
+ device_map={"": "cpu"},
)
That works! :) Thanks. Would be nice if it could run at least partly off GPU as it's just a bit slow but is working. Seems to take in the order of a minute or two I guess to answer questions.
(newenv) david@Davids-Mac-mini moondream % ./run.sh Fetching 12 files: 100%| 12/12 [00:00<00:00, 126144.48it/s] model_path=/Users/david/.cache/huggingface/hub/models--vikhyatk--moondream0/snapshots/c875d938535a26218a62d8a4c21818f3622dba6c
Describe the image The image features a red city bus driving down a street, surrounded by tall buildings. The bus is positioned in the middle of the scene, with a traffic light visible on the right side of the image. There are several cars on the street, with one car on the left side of the bus, another car in the middle, and two more cars on the right side of the scene. There are also multiple people on the street, with one person standing near the left side of the bus, another person closer to the middle, and two more people on the right side of the scene. A bicycle can be seen on the left
What color is the bus?
The bus is red.
How many people are inside the green bus? The image shows a red bus, not a train, driving down the street. However, there are several people inside the bus, indicating that it is a busy city street with a significant number of pedestrians.
What city does it look like if you had to guess? Based on the image, it appears that the city is New York. This is evident from the presence of a red passenger train traveling down the tracks, which is a common sight in New York City. Additionally, there are numerous cars and pedestrians on the street, suggesting that the city is bustling with activity. However, without more specific visual cues, it is difficult to pinpoint the exact city.
Agreed, it seems to work when it's fully on MPS or fully on CPU. Need to figure out why the disk offload is breaking it
Going to close this issue, we're no longer using disk offload in this repository.