[URGENT] llama.cpp / GGUF breaks on Colab

[danielhanchen]

llama.cpp seems to be unable to install - investigating now.

[eugeniosegala]

Facing the same issue.

Is Unsloth always pulling the latest version of llama.cpp? If yes, would not be safer to lock it? Happy to help with that.

[danielhanchen]

Sorry on the issue! Hmm I'm actually unsure what the issue is yet - some installs work, whilst some don't - I'm not sure what is causing it to fail

[danielhanchen]

@eugeniosegala Found the issue - reporting to HuggingFace. It seems like PEFT got an update, weirdly causing stuff to break in Colab.

Please change the installation instructions to

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.26" trl "peft<0.11.0" accelerate bitsandbytes

in the first cell

[kirilgorbachov]

Did not resolved the issue. I am getting:

Tried both: "peft<0.11.0"

And manual reinstallation to "peft==0.10.0"

Did not help. P.S. Yesterday everything was working fine. Some dependencies.

[kirilgorbachov]

When run for the first time, tries to re-install llama.cpp

[gaurav-nelson]

+1 Didn't resolve the issue. I still get the error TypeError: a bytes-like object is required, not 'str' in Colab notebook while running save_pretrained_gguf

[Jiar]

+1 Didn't resolve the issue. I still get the error TypeError: a bytes-like object is required, not 'str' in Colab notebook while running save_pretrained_gguf

the same as you

[danielhanchen]

@kirilgorbachov @Jiar @gaurav-nelson So so sorry it was my dumb mistake!! I accidentally did a typo!!

If you're still on Colab, please uninstall unsloth or edit the top cell to

# %%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip uninstall unsloth -y
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.26" trl "peft<0.11.0" accelerate bitsandbytes

then restart and run all

Sorry again!

[eugeniosegala]

@danielhanche, It's working now! Thanks 🙏

[gaurav-nelson]

Thank you @danielhanchen 🥇 It is working now.

[kirilgorbachov]

@danielhanchen I do not actually see your spelling mistake, but it defiantly fixed now and function properly.

I tried to debug it a bit yesterday, seems unsloth installs llama.cpp via git clone ... master of their repo. I actually believe that they had some issue for a while, and now it's resolved. Maybe versioning / release tagging would be a good idea.

@danielhanchen Thanks for your time anyway!

[danielhanchen]

As an update, the super speedy HF team have isolated the issue and have swiftly found a fix :) Hats off to them for their fabulous speed! https://github.com/huggingface/peft/pull/1739