Killed while loading pygmalion-6b_dev, GPT-J and other recent models

[HisashiASM]

Describe the bug

Returns Killed with no other useful information, only on specific (newer?) models.

Is there an existing issue for this?

• [x] I have searched the existing issues

Reproduction

Simply download latest pygmalion-6b_dev and try to load up. Please note this is a local machine and nothing hosted .

Screenshot

No response

Logs

:>python server.py --share

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/hisashiasm/miniconda3/envs/textgen did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/devkitpro/devkitARM')}
  warn(msg)
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/devkitpro/devkitPPC')}
  warn(msg)
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('unix')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Loading binary /home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
/home/hisashiasm/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
The following models are available:

1. opt-2.7b
2. pygmalion-6b_b8344bb4eb76a437797ad3b19420a13922aaabe1
3. pygmalion-6b_dev

Which one do you want to load? 1-3

3

Loading pygmalion-6b_dev...
Loading checkpoint shards:   0%|                                                                  | 0/2 [00:00<?, ?it/s]
Killed

System Info

CPU: AMD Ryzen 9 5900X
GPU: NVIDIA RTX 4090
RAM: 32GB DDR4 @3600MHz
System: Windows 11 (WSL Ubuntu)

[Ph0rk0z]

You run out of system ram?

[HisashiASM]

No, I watched my sys ram during the inload and it reached maybe 80 ish percent. Also why would older models work? The difference in model size is about 200MB

[Ph0rk0z]

Killed happened for me when OOM on linux. Is this WSL? Maybe it has some memory constraint. Also your bits and bytes is without GPU support. You don't have proper CUDA packages installed and your model is loading in FP32.

[HisashiASM]

You know, I have been banging my head in over why it isn't I have CUDA 11.7 installed on this VM, also why would I be using bits-and-bytes if I am not loading into 8bit

Update btw, pygmalion6b stable loads fine (so not the one from a while back)

[jllllll]

libcuda.so in WSL comes from Windows GPU drivers. There is a bug in WSL that causes drivers to not be properly set up in the virtual machine. Whether or not that is the issue, or if it is a bug in bits-and-bytes, I don't know. Either way, running this command before running server.py will get around this: export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH This command's effects are temporary and it needs to be run every time you want to use bits-and-bytes.

[Ph0rk0z]

Can you put it in .bashrc?

[jllllll]

Can you put it in .bashrc?

Yes.

[iChristGit]

libcuda.so in WSL comes from Windows GPU drivers. There is a bug in WSL that causes drivers to not be properly set up in the virtual machine. Whether or not that is the issue, or if it is a bug in bits-and-bytes, I don't know. Either way, running this command before running server.py will get around this: export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH This command's effects are temporary and it needs to be run every time you want to use bits-and-bytes.

I still get "Killed" after running your command. Fresh WSL install, folowing all the install steps one by one and just trying GPT J, ram is around 20GB out of 32GB?

[jllllll]

libcuda.so in WSL comes from Windows GPU drivers. There is a bug in WSL that causes drivers to not be properly set up in the virtual machine. Whether or not that is the issue, or if it is a bug in bits-and-bytes, I don't know. Either way, running this command before running server.py will get around this: export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH This command's effects are temporary and it needs to be run every time you want to use bits-and-bytes.

I still get "Killed" after running your command. Fresh WSL install, folowing all the install steps one by one and just trying GPT J, ram is around 20GB out of 32GB?

I've never used GPT-J with text-generation-webui before, I was only commenting on the WARNING! libcuda.so not found! part of the error log shown above. It might be worth trying this outside of WSL. I was finally able to get GPTQ-for-LLaMa to compile on Windows, and everything runs so much faster than inside of WSL.

[HisashiASM]

How much better would you say LLaMa is? Was curious of it's responses over GPT-J

[jllllll]

So far, it seems better at understanding code, but other than that it seems largely comparable to GPT-J. I heard there was an issue with sampler settings not working properly with LLaMa, so I'm not sure an accurate comparison can be made right now. As far as code is concerned, it's not that great, but it can surprise you occasionally. It almost wrote a complete program to interact with Steam Workshop utilizing Steamworks SDK. It also seems fairly adept at providing installation instructions for various software. All of my tests were with LLaMa loaded in 4bit, so that may be lowering the output quality on my end.

[razvan-nicolae]

retried with older pytorch - pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 still fails. Did not even go near the load-8-bit stuff yet, it flops. I also made a .wslconfig with a 24GB memory setting, absolutely zero difference. Usage is 22GB/32 for the system, VRAM no usage, and process gets killed. --- edit : mea culpa, loads fine after adding [wsl2] header to my wslconfig file, still, very weird it would not load with 16GB available to the WSL.

[jllllll]

@razvan-nicolae What version of WSL are you using?

[razvan-nicolae]

Wsl v2, ubuntu. All is well check my edit in the reply above

On 23 Mar 2023, at 06:55, jllllll @.***> wrote:



@razvan-nicolaehttps://github.com/razvan-nicolae What version of WSL are you using?

— Reply to this email directly, view it on GitHubhttps://github.com/oobabooga/text-generation-webui/issues/440#issuecomment-1480600412, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AESH55X5EE6NINCABS4CKH3W5PJQ3ANCNFSM6AAAAAAWAGXWP4. You are receiving this because you were mentioned.Message ID: @.***>

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

[sh4d0wph03n1x]

Still relevant. Having the same issue on bare metal Ubuntu with 24 gigs of RAM

[hokaso]

I also encountered the same problem, I am using a 3090 graphics card + 32G memory, no OOM

[mrtrizer]

You can see what happened dmesg -T| grep -E -i -B100 'killed process' It will likely say "Out of memory",

Create .wslconfig file in your windows user directory (C:\Users\smith) Make sure it doesn't have .txt extension Put memory limit setting there:

[wsl2]
memory=48GB

Details here: https://learn.microsoft.com/en-us/windows/wsl/wsl-config#wslconf

Last you need to restart your wsl wsl --shutdown

All this steps helped in my case

[jmiddleton]

On Amazon Linux I got the same issue. This is the log:

[Sun Mar 24 23:54:53 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=python,pid=6154,uid=1000 [Sun Mar 24 23:54:53 2024] Out of memory: Killed process 6154 (python) total-vm:18109076kB, anon-rss:14095848kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:29036kB oom_score_adj:0