RuntimeError: CUDA error: no kernel image is available for execution on the device

[amanjain1397appy]

I am deploying a model using torchserve. It was easily deployed on Tesla K80 GPU. But now when i shifted it to newer GPU Nvidia A30. I am getting this error

torch_version: '1.12.0+cu113' cudnn_version: 8302

PFA Nvidia-smi Screenshot

[msaroufim]

Hi @amanjain1397appy does your model also fail when you're not serving it with torchserve? I would need more repro instructions which you can see in the bug template we have here on github https://github.com/pytorch/serve/issues/new?assignees=&labels=&template=bug.yml

Regardless my suspicion is this is happening because we don't support pytorch 1.12 in torchserve yet but this was something @mreso had been asking for and @lxning was working on so we'll keep you posted!

If you run this script it should confirm which version of pytorch you actually have installed https://github.com/pytorch/serve/blob/master/ts_scripts/print_env_info.py

If you can't wait feel free to update the version for torch here in the requirements.txt and hopefully the issue just goes away https://github.com/pytorch/serve/tree/master/requirements

[amanjain1397appy]

@msaroufim My model does not fail when I am not serving with Torchserve. Speaking of the pytorch version, during my first attempt I actually used the torch version in requirements.txt and still I faced the same issue.

[msaroufim]

Hi @amanjain1397appy if you try torch version 1.11 without torchserve do you see this error?

[amanjain1397appy]

I don't get any such errors when I try torch 1.11 without torchserve.

[msaroufim]

Ok cool, if you can share a repro that'll help make things go faster. Otherwise we'll take a look at how we run on A100 when we upgrade to 1.12

[amanjain1397appy]

I'll try to share a repro.

Just wanted to let you know that when I try out the examples already present in torchserve repo for example dcgan_fashiongen they work absoutely fine.

Attaching this extra information just in case.

Even the current model in which I am facing issues, was working fine on Torchserve in other machine having the following specs.

[amanjain1397appy]

@msaroufim I have reproduced the code and pushed it here at https://github.com/amanjain1397appy/serve-personal