togethercomputer
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 432.00 MiB (GPU 2; 23.65 GiB total capacity; 20.88 GiB already allocated; 259.56 MiB free;
Describe the bug A clear and concise description of what the bug is.
To Reproduce Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
You ran out of GPU memory. Describe more on your setup like what you are using and what command you ran to resolve.
It'd be really cool if the minimum requirements of the model (size on disk for data set, vram requirements) on the readme, that would save a lot of people some time.
You ran out of GPU memory. Describe more on your setup like what you are using and what command you ran to resolve.
batch_size=4
It'd be really cool if the minimum requirements of the model (size on disk for data set, vram requirements) on the readme, that would save a lot of people some time.
That's a great idea. I'll put up a PR soon to document this.
(OpenChatKit) root@aca2869c8358:~/OpenChatKit-main# python inference/bot.py
Loading /root/OpenChatKit-main/inference/../huggingface_models/GPT-NeoXT-Chat-Base-20B to cuda:0...
Traceback (most recent call last):
File "/root/OpenChatKit-main/inference/bot.py", line 185, in
some problem, any idea how much memory it needs? or any solution to reduce the memory use? Thanks.
