"No GPU being used. Careful, inference might be very slow!"

[dreiscm]

I am running the code right now, and the audio file is in the proccess, but it is REALLY slow, like almost an hour. Along with the proccess, i got this message: "No GPU being used. Careful, inference might be very slow!". The thing is, i do have a NVIDA 1050 as GPU in my laptop. How can i make the programm use my GPU to run it faster?

[genetrader]

Same issue - I have a Nvidia GTX 2080 with 8gb of RAM.

[gkucsko]

hmm strange so two separate things

1. first check if torch can see your gpu. basically, try to get torch.cuda.is_available() running first independently of bark following their installs
2. even on cpu technically it should run faster than that. maybe try loading the smaller version of the models and see if that's faster?

def preload_models(
    text_use_small=True,
    coarse_use_small=True,
    fine_use_small=True,
)

[taktakut]

Hi, here is in my case. I had the same situation, and check Pytorch with print(torch.version) in my current environment. I found it's just for CPU (2.0.0+cpu), not for CUDA ver.

[rubinchyk]

RTX3070, same problem...

P.S. This helped https://github.com/suno-ai/bark/issues/173

[SECRET-GUEST]

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :

import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))

For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.

[dreiscm]

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :

import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))

For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.

Thank you for the help. It seems i did not have neither CUDA nor the the Torch Version to use the GPU. I fixed both of those, but now i am getting the following:

untyped_storage = torch.UntypedStorage( torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.45 GiB already allocated; 0 bytes free; 3.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[SECRET-GUEST]

Ok so you should probably enable small model, by changing False to True into bark/bark/generation.py

USE_SMALL_MODELS = os.environ.get("SUNO_USE_SMALL_MODELS", False) 

Also this interface is much better : https://github.com/C0untFloyd/bark-gui

[bdevel]

I received the "No GPU being used" warning because I'm using Apple silicon. Anyone with this error just needs to enable MPS with an environment variable. For example:

SUNO_ENABLE_MPS=True python3 speak.py

Note to maintainers, this should probably just be automatic if torch.backends.mps.is_available() == True

Everyone else, more info on the MacOS MPS backend for PyTorch.

[anjy077]

I also have this problem

[HWiese1980]

Same issue, you can check if you have a compatible version of torch with your graphical card while running in a python file .py :

import torch

print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Number of GPUs:", torch.cuda.device_count())
print("GPU name:", torch.cuda.get_device_name(0))

For me I'm compatible, so I suppose the problem come from the allocated Vram by default not settable.

Thank you for the help. It seems i did not have neither CUDA nor the the Torch Version to use the GPU. I fixed both of those, but now i am getting the following:

untyped_storage = torch.UntypedStorage( torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.45 GiB already allocated; 0 bytes free; 3.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Could you please tell us exactly what you did to solve your initial problem?

[HWiese1980]

Okay, I think I solved it too. What I did, in the corresponding virtualenv for bark:

bark$ pip3 install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

This should be enough to get GPU inference going.

[johlim]

cuda Download link https://developer.nvidia.com/cuda-downloads

• torch Download Link https://download.pytorch.org/whl/torch/ pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html

• tourchaudio tourch audio,vision downloadLink

https://download.pytorch.org/whl/torchaudio/ https://download.pytorch.org/whl/torchvision/

If you're using python 3.10 and torch 2.0, you can use torchaudio-2.0.0+cu117-cp310-cp310-win_amd64.whl torchvision-0.15.1+cpu-cp310-cp310-win_amd64.whl

---

• Install Example

 pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu117/torch_stable.html
Collecting torch==2.0.0+cu117
  Using cached https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl (2343.6 MB)
Requirement already satisfied: filelock in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.12.0)
Requirement already satisfied: typing-extensions in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (4.5.0)
Requirement already satisfied: sympy in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (1.12rc1)
Requirement already satisfied: networkx in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1)
Requirement already satisfied: jinja2 in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python310\lib\site-packages (from jinja2->torch==2.0.0+cu117) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\python310\lib\site-packages (from sympy->torch==2.0.0+cu117) (1.3.0)
Installing collected packages: torch
Successfully installed torch-2.0.0+cu117

D:\VDisk\Bark>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:59:34_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

D:\VDisk\Bark>python
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("Torch version:", torch.__version__)
Torch version: 2.0.0+cu117
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1

[johlim]

I was struggling with the out of memory error message, but when I typed it correctly in the running console environment, it worked. "set SUNO_USE_SMALL_MODELS=True"

    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 7.26 GiB already allocated; 0 bytes free; 7.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

D:\VDisk\Bark>set SUNO_USE_SMALL_MODELS=True

D:\VDisk\Bark>python main.py
100%|████████████████████████████████████████████████████████████████████████████████| 100/100 [00:01<00:00, 87.93it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 13/13 [00:03<00:00,  3.82it/s]

[mrbb]

cuda Download link https://developer.nvidia.com/cuda-downloads

• torch Download Link https://download.pytorch.org/whl/torch/ pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
• tourchaudio tourch audio,vision downloadLink

https://download.pytorch.org/whl/torchaudio/ https://download.pytorch.org/whl/torchvision/

If you're using python 3.10 and torch 2.0, you can use torchaudio-2.0.0+cu117-cp310-cp310-win_amd64.whl torchvision-0.15.1+cpu-cp310-cp310-win_amd64.whl

• Install Example

 pip install torch==2.0.0+cu117 -f https://download.pytorch.org/whl/cu117/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu117/torch_stable.html
Collecting torch==2.0.0+cu117
  Using cached https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl (2343.6 MB)
Requirement already satisfied: filelock in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.12.0)
Requirement already satisfied: typing-extensions in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (4.5.0)
Requirement already satisfied: sympy in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (1.12rc1)
Requirement already satisfied: networkx in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1)
Requirement already satisfied: jinja2 in c:\python310\lib\site-packages (from torch==2.0.0+cu117) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python310\lib\site-packages (from jinja2->torch==2.0.0+cu117) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\python310\lib\site-packages (from sympy->torch==2.0.0+cu117) (1.3.0)
Installing collected packages: torch
Successfully installed torch-2.0.0+cu117

D:\VDisk\Bark>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:59:34_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

D:\VDisk\Bark>python
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("Torch version:", torch.__version__)
Torch version: 2.0.0+cu117
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1

This has saved me a day. It works. Thank you.

[Niconord59]

Hello ! Is it possible to run it with an AMD 6900XT? Because although I have an AMD Ryzen 9 5950X it is slow.

[gkucsko]

right now amd is not supported and i can't test without an amd gpu. open to PRs from the community though if anyone can have a look

[rubinchyk]

right now amd is not supported and i can't test without an amd gpu. open to PRs from the community though if anyone can have a look

I have AMD and it's working for me with GPU.

[gkucsko]

looks like this is fixed for most people. also the topic of the issue is a bit non-descript since it just indicates lack of a gpu

[johlim]

@gkucsko It would be nice to add an example to the ReadMe documentation of using set SUNO_USE_SMALL_MODELS=True in the console before doing the python code.

[gkucsko]

yeah, since we didn't really include a command line script i figured from within python is easier, but ya if you launch something it could be useful to do outside. although now we have a preload_models option as well for small models which might be more straighforward for people anyways

[builder-main]

I also had to uninstall before reinstall as I installed pytorch before cuda 11.8 pip3 uninstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Before :

$ python info.py 
Torch version: 2.0.1+cpu
CUDA available: False
Number of GPUs: 0
AssertionError: Torch not compiled with CUDA enabled

After

$ python info.py 
Torch version: 2.0.1+cu118
CUDA available: True
Number of GPUs: 1
GPU name: NVIDIA GeForce RTX 3080

[JonathanFly]

Hello ! Is it possible to run it with an AMD 6900XT? Because although I have an AMD Ryzen 9 5950X it is slow.

It's not ROCM, but DirectML seems to be better than nothing for Window AMD:

https://github.com/JonathanFly/bark/issues/79#issuecomment-1597673112

https://github.com/JonathanFly/bark/tree/bark_amd_directml_test#-bark-amd-install-test-

[jaggzh]

I'm in Linux, but I'm getting lots of memory and cpu use, but not seeing any gpu use at all (unless it suddenly uses it for a split second at the end). Inference takes a long time -- like 6 minutes for a 24 word sentence.

>>> print("Torch version:", torch.__version__)
Torch version: 2.2.1+cu121
>>> print("CUDA available:", torch.cuda.is_available())
CUDA available: True
>>> print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 1
>>> print("GPU name:", torch.cuda.get_device_name(0))
GPU name: NVIDIA GeForce RTX 3090

$ time ./bark
2024-04-11 21:48:29.666824: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/.../python3.11/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
The attention mask and the pad token id were not set. As a consequence, you mayobserve unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:10000 for open-end generation.

real    5m57.385s
user    20m17.150s
sys     1m14.751s

[luke1212]

RTX3070, same problem...

P.S. This helped #173

This helps！ thanks