offload_dir error on Colab

[amrrs]

This error seems to be due to accelerate , curious if you have any solution?

Traceback (most recent call last):
  File "/content/alpaca-lora/generate.py", line 36, in <module>
    model = PeftModel.from_pretrained(
  File "/usr/local/lib/python3.9/dist-packages/peft/peft_model.py", line 177, in from_pretrained
    model = dispatch_model(model, device_map=device_map)
  File "/usr/local/lib/python3.9/dist-packages/accelerate/big_modeling.py", line 342, in dispatch_model
    raise ValueError(
ValueError: We need an `offload_dir` to dispatch this model according to this `device_map`, the following submodules need to be offloaded: base_model.model.model.layers.27, base_model.model.model.layers.28, base_model.model.model.layers.29, base_model.model.model.layers.30, base_model.model.model.layers.31, base_model.model.model.norm, base_model.model.lm_head.

[llcing]

Increasing the memory works for me.

[amrrs]

Did you try this on Google Colab?

[llcing]

Did you try this on Google Colab?

no, I tried on my own gpu server.

[amrrs]

Strangely this error seems to occur on Colab, not sure if it's because it's a 16 gig RAM @tloen Any thoughts?

[abdul-thoughtworks]

I also face the same error, using Colab !

[abdul-thoughtworks]

I also face the same error, using Colab !

Managed to address it by using a custom vm (using GCP) :)

[abdul-thoughtworks]

ror seems to occur on Colab, not sure if it's because it's a 16 gig

Yes it is the memory I believe, Once I plugged in a custom vm with 30gigs it worked fine.

[zhangbo2008]

i only can run correctly on my without Gpu server