cupscale icon indicating copy to clipboard operation
cupscale copied to clipboard

Unable to upscale full image because of time out

Open RunDevelopment opened this issue 3 years ago • 4 comments

I am unable to preview a full-image upscale. When upscaling a 1024x1024 PNG using the "Refresh preview (full image)" button, I get a generic error message after about 2 minutes that an error occurred. Looking at the log, it seems like Cupscale or the python process has a time out of around 2 minutes or 100 seconds, after which it will throw an error.

This error does not occur on my desktop PC (Nvidia GTX 1080Ti), it only happens on my laptop (Nvidia GTX 950M). I suspect that the mobile GPU of my laptop is so slow that Cupscale simply declares the upscaling to have failed when I can see that the CUDA cores of my GPU are still doing work (in TaskManager).

Is there any way to increase the timeout?

Full log
Running in portable mode. Data folder: C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData
Arg: C:\DS3TexUp\Cupscale 1.39.0f1\Cupscale.exe
Detected Card: NVIDIA GeForce GTX 950M / GPU: GM107-A / Arch: Maxwell
Initialized Nvidia API. GPU: NVIDIA GeForce GTX 950M
[DepCheck] CMD: /C python -V
[DepCheck] System Python Check Output: Python 3.9.9
[DepCheck] Sys Python Ver: Python 3.9.9
[Servers] Ping to nmkd-hz.de: 37 ms
Parsing Patrons from CSV...
Found 14 Gold Patrons, 406 Silver Patrons
[Servers] Ping to nmkd-cb.de: 200 ms
[Servers] Closest Server: Germany (Nürnberg) (37 ms)
[MainUI] Dropped 1 file(s), first = C:\DS3TexUp\upscale\m31\a\m31_00_woodplank_10_a.png
[ImgProc] Converting C:\DS3TexUp\upscale\m31\a\m31_00_woodplank_10_a.png to PngRaw, DelSrc: False, Fill: True, Ext: UseNew
[ImgProc] Color depth of m31_00_woodplank_10_a.png is 32.
[ImgProc] Written image to C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\loaded-img\temp.png
[ImgProc] Preprocessing C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview\preview.png - Fill Alpha: True
[ImgProc] Color depth of preview.png is 24.
[ImgProc] Done pre-processing images
[CMD] /C cd /D "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch" & python upscale.py --input "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview" --output "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview-out"   --device_id 0 --fp16  --alpha_mode 0   "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth;50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth;50"
[Python] Traceback (most recent call last):
[Python] Model: 4x_UniversalUpscalerV2-Neutral_115000_swaG
[Python] Upscaling...
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 31, in auto_split_upscale
[Python] 1 preview.png.png
[Python]     result = upscale_function(lr_img)
[Python] Image loaded
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 282, in upscale
[Python] Reloading model - model_path: C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth@50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth@50 - last_model: None
[Python] Setting last_model to C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth@50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth@50
[Python] Upscaling image...
[Python]     output = process(img1)
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 126, in process
[Python]     output = model(img_LR).data.squeeze(
[Python] RuntimeError: CUDA error: the launch timed out and was terminated
[Python] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[Python] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[Python] During handling of the above exception, another exception occurred:
[Python] Traceback (most recent call last):
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 375, in <module>
[Python]     rlt, depth, _ = ops.auto_split_upscale(img, upscale, last_scale)
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 55, in auto_split_upscale
[Python]     top_left_rlt, depth, current_tile_num = auto_split_upscale(top_left, upscale_function, scale=scale, overlap=overlap, current_depth=current_depth+1, current_tile_num=current_tile_num)
[Python]   File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 43, in auto_split_upscale
[Python]     raise RuntimeError(e)
[Python] RuntimeError: CUDA error: the launch timed out and was terminated
[Python] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[Python] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[ESRGAN] Upscaling Error: Der Index war außerhalb des Arraybereichs.
   bei Cupscale.Main.Upscale.<Run>d__8.MoveNext()

RunDevelopment avatar Dec 21 '21 18:12 RunDevelopment

There is no such timeout, I'd say the resolution is just too high for that GPU

n00mkrad avatar Dec 22 '21 21:12 n00mkrad

I can somewhat confirm that guess. I noticed that 512x512 images did not work, but 400x400 images did.

However, isn't Cupscale supposed to automatically tile large images? It had no problem with upscaling large images (4096x4096) using tiles on my desktop PC. (Cupscale does tile 1024x1024 images but it seems to not tile 512x512 images on my laptop.)

RunDevelopment avatar Dec 22 '21 22:12 RunDevelopment

I guess it depends on your available RAM/VRAM....?

twisted-nematic57 avatar Dec 25 '21 06:12 twisted-nematic57

My GTX 950M has 4GB of VRAM, Cupscale uses about 2.5GB of that. My system has about 4GB RAM left available when running Cupscale, and it uses about 1.7GB of that.

RunDevelopment avatar Dec 25 '21 10:12 RunDevelopment