GPU not utilized

[stavsap]

Hi,

i am running with 3090 but it seems still it uses CPU.

in the log i see "Running on: cuda" but examining the resource usage the GPU is almost untouched.

what might be the issue?

i am running with "pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118"

[lelapin123]

try to see if replacing "device_map=1" by "device_map='auto'" changes something. With my version it returns then an out of memory error (but it uses the GPU)

[Tchekda]

With my version it returns then an out of memory error (but it uses the GPU)

@lelapin123 try adding max_memory={0: "15GB"} to the AutoModelForCausalLM.from_pretrained function, like this :

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='auto',
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    max_memory={0: "15GB"}
)

I added it in #139

[stavsap]

seems to utilize GPU now, thanks

[anmolbyte]

sorry, where did you replace "device_map=1" by "device_map='auto'? I'm running into the same issue

[yirunwang]

@stavsap @ajarang I have same issue, please advise how can I get it work? thanks

[anmolbyte]

this thread is outdated. Check to see if you have Cuda, llama, and visual studio cuda extensions installed properly

[Tobias-Schnier]

If anyone stumbles on this topic and is on Windows, this is the solution: After you install all requirements, you have to explicitely install torch with cuda! That means do

pip3 uninstall torch torchvision torchaudio

and then

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 .

Make sure to use the right version of cuda for your graphics card!