InvokeAI
InvokeAI copied to clipboard
invoke-ai
[bug]: Fails to generate an image - Windows 10 - RuntimeError: CUDA error: device-side assert triggered
Is there an existing issue for this?
- [X] I have searched the existing issues
OS
Windows
GPU
cuda
VRAM
24
What happened?
after a long long manual install since the binary one just wont work for some reason I was able to get all the required packages set up from the env, base and cuda requirements. I can launch the app and load it from my local lan but anytime I try to create an image it bails on RuntimeError: CUDA error: device-side assert triggered.
(invokeai) PS C:\Users\evil\Desktop\Work\invokeAI> python .\scripts\invoke.py --web --host 192.168.0.109
>> Patchmatch initialized
* Initializing, be patient...
>> Initialization file C:\Users\evil/.invokeai found. Loading...
>> InvokeAI runtime directory is "C:\Users\evil\Desktop\Work\invokeAI"
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type cuda
>> Current VRAM usage: 0.00G
>> Scanning Model: stable-diffusion-1.5
>> Model Scanned. OK!!
>> Loading stable-diffusion-1.5 from C:\Users\evil\Desktop\Work\invokeAI\models\ldm\stable-diffusion-v1\v1-5-pruned-emaonly.ckpt
| LatentDiffusion: Running in eps-prediction mode
| DiffusionWrapper has 859.52 M params.
| Making attention of type 'vanilla' with 512 in_channels
| Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
| Making attention of type 'vanilla' with 512 in_channels
| Using faster float16 precision
| Loading VAE weights from: C:\Users\evil\Desktop\Work\invokeAI\models\ldm\stable-diffusion-v1\vae-ft-mse-840000-ema-pruned.ckpt
>> Model loaded in 10.08s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Current embedding manager terms: *
>> Setting Sampler to k_euler_a
* --web was specified, starting web server...
>> Initialization file C:\Users\evil/.invokeai found. Loading...
>> Started Invoke AI Web Server!
>> Default host address now 127.0.0.1 (localhost). Use --host 0.0.0.0 to bind any address.
>> Point your browser at http://192.168.0.109:9090
>> System config requested
>> System config requested
>> System config requested
>> Image generation requested: {'prompt': 'A full body photo-real delicate sculpture of an ornate detailed fully nude female warrior princess in front of a intricate fantasy nighttime background, beautiful eyes, micro detail, backlit lighting, nice breasts, armor encrusted with gems, colorful, physically based rendering, tribal art, trending on cgsociety, beautiful face, smooth, 8k, small ears, centered, photorealistic, mdjrny-v4 style, detailed image, nice smile, beautiful eyes, quixel megascans, full body portrait ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_lms', 'seed': 4241787346, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '...', 'seamless': False, 'hires_fix': False, 'variation_amount': 0}
ESRGAN parameters: False
Facetool parameters: False
{'prompt': 'A full body photo-real delicate sculpture of an ornate detailed fully nude female warrior princess in front of a intricate fantasy nighttime background, beautiful eyes, micro detail, backlit lighting, nice breasts, armor encrusted with gems, colorful, physically based rendering, tribal art, trending on cgsociety, beautiful face, smooth, 8k, small ears, centered, photorealistic, mdjrny-v4 style, detailed image, nice smile, beautiful eyes, quixel megascans, full body portrait ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_lms', 'seed': 4241787346, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '', 'seamless': False, 'hires_fix': False, 'variation_amount': 0}
>> Setting Sampler to k_lms
Generating: 0%| | 0/1 [00:00<?, ?it/s]>> Ksampler using model noise schedule (steps >= 30)
>> Sampling with k_lms starting at step 0 of 50 (50 new sampling steps)
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [70,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
0%| | 0/50 [00:02<?, ?it/s]
Generating: 0%| | 0/1 [00:02<?, ?it/s]
Traceback (most recent call last):
File "c:\users\evil\desktop\work\invokeai\ldm\generate.py", line 488, in prompt2image
results = generator.generate(
File "c:\users\evil\desktop\work\invokeai\ldm\invoke\generator\base.py", line 98, in generate
image = make_image(x_T)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\invoke\generator\txt2img.py", line 41, in make_image
samples, _ = sampler.sample(
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\ksampler.py", line 223, in sample
K.sampling.__dict__[f'sample_{self.schedule}'](
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\k_diffusion\sampling.py", line 267, in sample_lms
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\ksampler.py", line 52, in forward
next_x = self.invokeai_diffuser.do_diffusion_step(x, sigma, uncond, cond, cond_scale)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\shared_invokeai_diffusion.py", line 107, in do_diffusion_step
unconditioned_next_x, conditioned_next_x = self.apply_standard_conditioning(x, sigma, unconditioning, conditioning)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\shared_invokeai_diffusion.py", line 123, in apply_standard_conditioning
unconditioned_next_x, conditioned_next_x = self.model_forward_callback(x_twice, sigma_twice,
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\ksampler.py", line 38, in <lambda>
model_forward_callback=lambda x, sigma, cond: self.inner_model(x, sigma, cond=cond))
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\k_diffusion\external.py", line 114, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\k_diffusion\external.py", line 140, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\ddpm.py", line 1441, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\ddpm.py", line 2167, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\diffusionmodules\openaimodel.py", line 806, in forward
h = module(h, emb, context)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\diffusionmodules\openaimodel.py", line 88, in forward
x = layer(x, context)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\attention.py", line 271, in forward
x = block(x, context=context)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\attention.py", line 221, in forward
return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\diffusionmodules\util.py", line 159, in checkpoint
return func(*inputs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\attention.py", line 226, in _forward
x += self.attn2(self.norm2(x.clone()), context=context)
File "C:\Users\evil\anaconda3\envs\invokeai\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "c:\users\evil\desktop\work\invokeai\ldm\modules\attention.py", line 199, in forward
r = self.get_invokeai_attention_mem_efficient(q, k, v)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\cross_attention_control.py", line 291, in get_invokeai_attention_mem_efficient
return self.einsum_op_cuda(q, k, v)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\cross_attention_control.py", line 285, in einsum_op_cuda
return self.einsum_op_tensor_mem(q, k, v, mem_free_total / 3.3 / (1 << 20))
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\cross_attention_control.py", line 264, in einsum_op_tensor_mem
return self.einsum_lowest_level(q, k, v, None, None, None)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\cross_attention_control.py", line 229, in einsum_lowest_level
self.attention_slice_calculated_callback(attention_slice, dim, offset, slice_size)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\shared_invokeai_diffusion.py", line 69, in <lambda>
lambda slice, dim, offset, slice_size, key=key: callback(slice, dim, offset, slice_size, key))
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\shared_invokeai_diffusion.py", line 61, in callback
saver.add_attention_maps(slice, key)
File "c:\users\evil\desktop\work\invokeai\ldm\models\diffusion\cross_attention_map_saving.py", line 39, in add_attention_maps
self.collated_maps[key_and_size] += maps.cpu()
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
>> Could not generate image.
>> Usage stats:
>> 0 image(s) generated in 2.86s
>> Max VRAM used for this generation: 3.29G. Current VRAM utilization: 2.17G
>> Max VRAM used since script start: 3.29G
conda list
# packages in environment at C:\Users\evil\anaconda3\envs\invokeai:
#
# Name Version Build Channel
absl-py 1.3.0 pypi_0 pypi
accelerate 0.15.0 pypi_0 pypi
addict 2.4.0 pypi_0 pypi
aiohttp 3.8.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
albumentations 1.3.0 pypi_0 pypi
altair 4.2.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 22.1.0 pypi_0 pypi
basicsr 1.4.2 pypi_0 pypi
bidict 0.22.0 pypi_0 pypi
blas 1.0 mkl
blinker 1.5 pypi_0 pypi
boltons 21.0.0 pypi_0 pypi
brotlipy 0.7.0 py310h2bbff1b_1002
bzip2 1.0.8 he774522_0
ca-certificates 2022.10.11 haa95532_0
cachetools 5.2.0 pypi_0 pypi
certifi 2022.9.24 py310haa95532_0
cffi 1.15.1 py310h2bbff1b_3
chardet 4.0.0 pypi_0 pypi
charset-normalizer 2.0.4 pyhd3eb1b0_0
clean-fid 0.1.34 pypi_0 pypi
click 8.1.3 pypi_0 pypi
clip 1.0 pypi_0 pypi
clipseg 0.0.1 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
contourpy 1.0.6 pypi_0 pypi
cryptography 38.0.1 py310h21b164f_0
cuda 11.6.2 0 nvidia
cuda-cccl 11.6.55 0 nvidia
cuda-command-line-tools 11.6.2 0 nvidia
cuda-compiler 11.6.2 0 nvidia
cuda-cudart 11.6.55 0 nvidia
cuda-cudart-dev 11.6.55 0 nvidia
cuda-cuobjdump 11.6.124 0 nvidia
cuda-cupti 11.6.124 0 nvidia
cuda-cuxxfilt 11.6.124 0 nvidia
cuda-libraries 11.6.2 0 nvidia
cuda-libraries-dev 11.6.2 0 nvidia
cuda-memcheck 11.8.86 0 nvidia
cuda-nsight-compute 12.0.0 0 nvidia
cuda-nvcc 11.6.124 0 nvidia
cuda-nvdisasm 12.0.76 0 nvidia
cuda-nvml-dev 11.6.55 0 nvidia
cuda-nvprof 12.0.90 0 nvidia
cuda-nvprune 11.6.124 0 nvidia
cuda-nvrtc 11.6.124 0 nvidia
cuda-nvrtc-dev 11.6.124 0 nvidia
cuda-nvtx 11.6.124 0 nvidia
cuda-nvvp 12.0.90 0 nvidia
cuda-runtime 11.6.2 0 nvidia
cuda-sanitizer-api 12.0.90 0 nvidia
cuda-toolkit 11.6.2 0 nvidia
cuda-tools 11.6.2 0 nvidia
cuda-visual-tools 11.6.2 0 nvidia
cycler 0.11.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
dependency-injector 4.40.0 pypi_0 pypi
diffusers 0.9.0 pypi_0 pypi
dnspython 2.2.1 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
einops 0.6.0 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
eventlet 0.33.2 pypi_0 pypi
facexlib 0.2.5 pypi_0 pypi
filelock 3.8.2 pypi_0 pypi
filterpy 1.4.5 pypi_0 pypi
flask 2.1.3 pypi_0 pypi
flask-cors 3.0.10 pypi_0 pypi
flask-socketio 5.3.0 pypi_0 pypi
flaskwebgui 0.3.7 pypi_0 pypi
flit-core 3.6.0 pyhd3eb1b0_0
fonttools 4.38.0 pypi_0 pypi
freetype 2.12.1 ha860e81_0
frozenlist 1.3.3 pypi_0 pypi
fsspec 2022.11.0 pypi_0 pypi
ftfy 6.1.1 pypi_0 pypi
future 0.18.2 py310haa95532_1
getpass-asterisk 1.0.1 pypi_0 pypi
gfpgan 1.3.8 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.29 pypi_0 pypi
google-auth 2.15.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
greenlet 2.0.1 pypi_0 pypi
grpcio 1.51.1 pypi_0 pypi
huggingface-hub 0.11.1 pypi_0 pypi
idna 2.10 pypi_0 pypi
imageio 2.22.4 pypi_0 pypi
imageio-ffmpeg 0.4.7 pypi_0 pypi
importlib-metadata 5.1.0 pypi_0 pypi
intel-openmp 2021.4.0 haa95532_3556
invokeai 2.2.0 dev_0 <develop>
itsdangerous 2.1.2 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.2.0 pypi_0 pypi
jpeg 9e h2bbff1b_0
jsonmerge 1.9.0 pypi_0 pypi
jsonschema 4.17.3 pypi_0 pypi
k-diffusion 0.0.1 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
kornia 0.6.8 pypi_0 pypi
lerc 3.0 hd77b12b_0
libcublas 12.0.1.189 0 nvidia
libcublas-dev 12.0.1.189 0 nvidia
libcufft 11.0.0.21 0 nvidia
libcufft-dev 11.0.0.21 0 nvidia
libcurand 10.3.1.50 0 nvidia
libcurand-dev 10.3.1.50 0 nvidia
libcusolver 11.4.2.57 0 nvidia
libcusolver-dev 11.4.2.57 0 nvidia
libcusparse 12.0.0.76 0 nvidia
libcusparse-dev 12.0.0.76 0 nvidia
libdeflate 1.8 h2bbff1b_5
libffi 3.4.2 hd77b12b_6
libnpp 12.0.0.30 0 nvidia
libnpp-dev 12.0.0.30 0 nvidia
libnvjpeg 12.0.0.28 0 nvidia
libnvjpeg-dev 12.0.0.28 0 nvidia
libpng 1.6.37 h2a8f88b_0
libtiff 4.4.0 h8a3f274_2
libuv 1.40.0 he774522_0
libwebp 1.2.4 h2bbff1b_0
libwebp-base 1.2.4 h2bbff1b_0
llvmlite 0.39.1 pypi_0 pypi
lmdb 1.4.0 pypi_0 pypi
lz4-c 1.9.3 h2bbff1b_1
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.6.2 pypi_0 pypi
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py310h2bbff1b_0
mkl_fft 1.3.1 py310ha0764ea_0
mkl_random 1.2.2 py310h4ed8f06_0
multidict 6.0.3 pypi_0 pypi
networkx 2.8.8 pypi_0 pypi
ninja 1.10.2 haa95532_5
ninja-base 1.10.2 h6d14046_5
nsight-compute 2022.4.0.15 0 nvidia
numba 0.56.4 pypi_0 pypi
numpy 1.23.3 py310h60c9a35_0
numpy-base 1.23.3 py310h04254f7_0
oauthlib 3.2.2 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
opencv-python 4.6.0.66 pypi_0 pypi
opencv-python-headless 4.6.0.66 pypi_0 pypi
openssl 1.1.1s h2bbff1b_0
packaging 22.0 pypi_0 pypi
pandas 1.5.2 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
picklescan 0.0.5 pypi_0 pypi
pillow 9.2.0 py310hdc2b20a_1
pip 22.2.2 py310haa95532_0
promise 2.3 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.4 pypi_0 pypi
pudb 2014.1 pypi_0 pypi
pyarrow 10.0.1 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pydeck 0.8.0 pypi_0 pypi
pydeprecate 0.3.2 pypi_0 pypi
pygments 2.13.0 pypi_0 pypi
pympler 1.0.1 pypi_0 pypi
pyopenssl 22.0.0 pyhd3eb1b0_0
pyparsing 3.0.9 pypi_0 pypi
pypatchmatch 0.1.4 pypi_0 pypi
pyreadline3 3.4.1 pypi_0 pypi
pyrsistent 0.19.2 pypi_0 pypi
pysocks 1.7.1 py310haa95532_0
python 3.10.8 h966fe2a_1
python-dateutil 2.8.2 pypi_0 pypi
python-engineio 4.3.4 pypi_0 pypi
python-socketio 5.7.2 pypi_0 pypi
pytorch-cuda 11.6 h867d48c_0 pytorch
pytorch-lightning 1.7.7 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pytz 2022.6 pypi_0 pypi
pytz-deprecation-shim 0.1.0.post0 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 py310h2bbff1b_1
qudida 0.0.4 pypi_0 pypi
realesrgan 0.3.0 pypi_0 pypi
regex 2022.10.31 pypi_0 pypi
requests 2.25.1 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
resize-right 0.0.2 pypi_0 pypi
rich 12.6.0 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.2.0 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
semver 2.13.0 pypi_0 pypi
send2trash 1.8.0 pypi_0 pypi
sentry-sdk 1.11.1 pypi_0 pypi
setproctitle 1.3.2 pypi_0 pypi
setuptools 65.5.0 py310haa95532_0
shortuuid 1.0.11 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_1
smmap 5.0.0 pypi_0 pypi
sqlite 3.40.0 h2bbff1b_0
streamlit 1.15.2 pypi_0 pypi
taming-transformers-rom1504 0.0.6 pypi_0 pypi
tb-nightly 2.12.0a20221210 pypi_0 pypi
tensorboard 2.11.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
test-tube 0.7.5 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2022.10.10 pypi_0 pypi
tk 8.6.12 h2bbff1b_0
tokenizers 0.12.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
torch 1.12.1+cu116 pypi_0 pypi
torch-fidelity 0.3.0 pypi_0 pypi
torchaudio 0.12.1+cu116 pypi_0 pypi
torchdiffeq 0.2.3 pypi_0 pypi
torchmetrics 0.11.0 pypi_0 pypi
torchsde 0.2.5 pypi_0 pypi
torchvision 0.13.1+cu116 pypi_0 pypi
tornado 6.2 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
trampoline 0.1.2 pypi_0 pypi
transformers 4.21.3 pypi_0 pypi
typing-extensions 4.4.0 py310haa95532_0
typing_extensions 4.4.0 py310haa95532_0
tzdata 2022.7 pypi_0 pypi
tzlocal 4.2 pypi_0 pypi
urllib3 1.26.13 py310haa95532_0
urwid 2.1.2 pypi_0 pypi
validators 0.18.2 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wandb 0.13.6 pypi_0 pypi
watchdog 2.2.0 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
whichcraft 0.6.1 pypi_0 pypi
win_inet_pton 1.1.0 py310haa95532_0
wincertstore 0.2 py310haa95532_2
xz 5.2.8 h8cc25b3_0
yaml 0.2.5 he774522_0
yapf 0.32.0 pypi_0 pypi
yarl 1.8.2 pypi_0 pypi
zipp 3.11.0 pypi_0 pypi
zlib 1.2.13 h8cc25b3_0
zstd 1.5.2 h19a0ad4_0
(invokeai) PS C:\Users\evil\Desktop\Work\invokeAI\environments-and-requirements>```
Kinda given up at this time, I was pretty excited about this but it seems like its just not in my cards for some reason :/
thoughts?
btw my machine is set up for cuda 11.8 on a Titan RTX gpu.. and all other things run fine that I install.. Auto1111, custom inpainting, etc.. its basically my AI machine that I run a lot from conda envs
### Screenshots
_No response_
### Additional context
_No response_
### Contact Details
_No response_
I want to add to this that my environment was working fine until the most recent pull. Now I receive the same error as above, so this appears to be an issue with one of the most recent changes.
Refreshing the environment with the install script does not resolve the issue, nor does a reboot.
Happened to me as well on latest commit (8cbb50c204b7ae24f805574aa59ede1ed2156318) only if I tried generating an image with a long prompt such as list of keywords obtained from DeepDanbooru-based interrogator.
I have the same symptoms, but different log!
The interesting part is:
> Ksampler using model noise schedule (steps >= 30)
>> Sampling with k_lms starting at step 0 of 50 (50 new sampling steps)
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [408,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
The offending prompt is: High detail RAW color sensual photo of ( Pare 25yo woman) with perfect face and perfect bright eyes, pale skin and freckles, large eyes, (wearing robe) (cone hair bun) realistic, highly detailed, harsh lighting, cinematic lighting, art by artgerm and greg rutkowski and alphonse mucha, (performing magic with fireflies, embers swirling around her) hasselblad, 45 degree, hard light, gigapixel. Cutting it approximately by half fixes the issue. Tell me if you need a precise value.
Win 10, GTX3070Ti, 8GB VRAM. Everything updated.
I get the same error
System config requested System config requested Image generation requested: {'prompt': 'A figure stands in a surreal landscape of rolling hills, strange trees and a pink sea. ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_euler_a', 'seed': 1414890566, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '...', 'seamless': False, 'hires_fix': True, 'variation_amount': 0} ESRGAN parameters: False Facetool parameters: {'type': 'gfpgan', 'strength': 0.8} {'prompt': 'A figure stands in a surreal landscape of rolling hills, strange trees and a pink sea. ', 'iterations': 1, 'steps': 50, 'cfg_scale': 7.5, 'threshold': 0, 'perlin': 0, 'height': 512, 'width': 512, 'sampler_name': 'k_euler_a', 'seed': 1414890566, 'progress_images': False, 'progress_latents': True, 'save_intermediates': 5, 'generation_mode': 'txt2img', 'init_mask': '', 'seamless': False, 'hires_fix': True, 'variation_amount': 0} Traceback (most recent call last): File "c:\invoke\invokeai.venv\lib\site-packages\ldm\generate.py", line 462, in prompt2image uc, c, extra_conditioning_info = get_uc_and_c_and_ec( File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 24, in get_uc_and_c_and_ec conditioning = _get_conditioning_for_prompt(prompt, negative_prompt, model, log_tokens) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 100, in _get_conditioning_for_prompt conditioning, _ = _get_embeddings_and_tokens_for_prompt(model, File "c:\invoke\invokeai.venv\lib\site-packages\ldm\invoke\conditioning.py", line 224, in _get_embeddings_and_tokens_for_prompt embeddings, tokens = model.get_learned_conditioning([fragments], return_tokens=True, fragment_weights=[weights]) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\models\diffusion\ddpm.py", line 829, in get_learned_conditioning c = self.cond_stage_model.encode( File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 461, in encode return self(text, **kwargs) File "c:\invoke\invokeai.venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 506, in forward tokens, per_token_weights = self.get_tokens_and_weights(fragments, weights) File "c:\invoke\invokeai.venv\lib\site-packages\ldm\modules\encoders\modules.py", line 634, in get_tokens_and_weights all_tokens_tensor = torch.tensor(all_tokens, dtype=torch.long).to(self.device) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Could not generate image. Usage stats: 0 image(s) generated in 0.03s Max VRAM used for this generation: 2.17G. Current VRAM utilization: 2.17G Max VRAM used since script start: 3.29G
Same problem.. Works ok when i am using short prompt, but long prompt causes problem.. Update or restart doesn`t help
anyone have a fix? Any devs check this out?
I can also verify it works with shorter prompts..
Having the same problem. For me however it appears to occur when the prompt has been truncated more than 1 token (ie. truncate 1 - generates ok, truncate 2 or more and generates an error.)
Example Steps: 25, CFG: 7.5, Size: 1024x1024, Sampler: k_lms
-
FINE: No Truncated Prompt an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive
-
FINE: truncates 1 prompt token an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive, closeup
-
ERROR: truncates 2 prompt tokens an (ultra-realistic)+ (full length portrait)+ of a futuristic (mechanical)+ female robot, looks like a celebrity, cyberpunk style, (frontal)+, (standing)++, (centered)+, (stretching)++, (full shot)+, full length, feet, elegant, (highly detailed)++, intricate, tone mapped, (Photo)++, (Realistic)+++, Canon50, (HD)++, (detailed)+, smooth, sharp focus, colorful, blade runner style city background, attractive, closeup face
This required completely closing the application and restarting to continue with InvokeAI
Decided to take a closer look at this since there seems to be no traction in getting this fixed.
The original issue appears to be caused by commit 786b8878d6c6a7c8db241e340c4551e71c1d6f15. Doing a git reset --hard on the commit before this resolves all issues.
This commit does generate a different error that is related to a fix in 12a8d7fc1494c24ba42fafe6c8e22249961d3bc9 once that fix is applied the current error is presented.
The 786b8878d6c6a7c8db241e340c4551e71c1d6f15 commit shows a lot of different files being updated, not sure yet where the actual issue is. Looking at #1866 I do see other issue the commit caused already being fixed.
cross_attention_maps_saving.py seems to be where the code dies off for me. Specifically when calling add_attention_maps.
When I use the first prompt from malbathla, I get a tokens_id range of (1,73) and everything works properly. When I use the third prompt I get tokens_id range of (1,78)
The code does mention that the range should typically be 77, so I am guessing there should be some sort of split there if there are more tokens in the prompt.
When the end range goes above 77 the maps = maps[:,:, self.token_ids] call causes the index out of bounds error.
As a work around, to just get stuff working, on the latest commit. If you add a return directly after the setup_attention_map_saving callback definition in shared_invokeai_diffusion.py everything seems to work as expected: ~ line 56:
def setup_attention_map_saving(self, saver: AttentionMapSaver):
def callback(slice, dim, offset, slice_size, key):
return
Can confirm that this work around works well and I can now generate images regardless of prompt size - nice. Given that this is a system breaking problem I am surprised it hasn't been addessed and fixed - thank goodness for community I guess.
Hey all - Monitoring issue. Unable to recreate it on my side, but recognize this bit is causing issues somewhere.
For now, my recommendation would be to use prompts that fit within the token limit, and take the silver lining view that this helps you identify when your prompting is going to waste 💁♂️
Thanks for the digging @JamesDooley
AH HA.
Can folks confirm that this is only happening on the unified canvas? I.e., it doesn't have issues on the Txt2Img tab?
When I run text to image, I get the same issue, but I do see an additional message about the prompt being too long and that it was truncated. It still errors out with index out of bounds.
>> Setting Sampler to k_euler_a
>> Prompt is 2 token(s) too long and has been truncated
Generating: 0%| | 0/12 [00:00<?, ?it/s]>> Ksampler using model noise schedule (steps >= 30)
>> Sampling with k_euler_ancestral starting at step 0 of 40 (40 new sampling steps)
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [27,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [108,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
All of my previous tests were against the unified canvas.
Correct, I commented out the early return so it was running the original code.
any pushed fix for this yet? Asking us to do smaller prompts is not a solution, and errors should always bee handled gracefully .. I know their is a lot going on with this amazing product but just curious where this currently stands as fixed
@lstein can you cut a release for this? Getting this at the moment The automatic installer doesn't seem to be updated yet, but you can work around it by doing this: Go to: https://github.com/invoke-ai/InvokeAI/tree/main/ldm/invoke Copy the file or the content of "conditioning.py" and either replace or copy and paste the content into your conditioning.py located in your invokeai folder > venv\Lib\site-packages\ldm\invoke
