LocalAI
LocalAI copied to clipboard
Unable to generate sd3 medium images with 24gb gpu
LocalAI version: localai/localai:v2.17.1-cublas-cuda12
Environment, CPU architecture, OS, and Version: Linux sphinx 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux RTX 4090
Describe the bug When trying to generate a 512x512 or 256x256 image it runs the 24gb gpu out of memory
To Reproduce
curl http://localhost:8080/v1/images/generations
-H "Content-Type: application/json"
-d '{
"prompt": "a test image",
"model": "stable-diffusion-3-medium",
"size": "256x256"
}'
Expected behavior generating an image
Logs
10:29PM ERR Server error error="could not load model (no success): Unexpected err=RuntimeError('CUDA error: out of memory\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with TORCH_USE_CUDA_DSA
to enable device-side assertions.\n'), type(err)=<class 'RuntimeError'>"