automatic
automatic copied to clipboard
Published
20 hours ago •
vladmandic
vladmandic
[Issue]: Inpainting on Intel Arc crashes
Issue Description
I tried to inpaint a picture using ipex on an Intel Arc 750. However it crashes right in the Intel libs again.
Stack Trace:
Stack trace of thread 24971:
#0 0x00007034a18ac83c n/a (libc.so.6 + 0x8e83c)
#1 0x00007034a185c668 raise (libc.so.6 + 0x3e668)
#2 0x00007034a18444b8 abort (libc.so.6 + 0x264b8)
#3 0x000070337de870a7 n/a (libze_intel_gpu.so.1 + 0x870a7)
#4 0x000070337e362bc9 n/a (libze_intel_gpu.so.1 + 0x562bc9)
#5 0x000070337df5a7a3 n/a (libze_intel_gpu.so.1 + 0x15a7a3)
#6 0x000070338fa1795c n/a (/home/sseifried/vladmandic-webui/venv/lib/libpi_level_zero.so + 0x8995c)
ELF object binary architecture: AMD x86-64
Looking at the log output from webui.sh, I see two suspicious assertions right before the crash (see relevant log output).
I already did disable all extensions, and I can confirm that I can generate images just fine using txt2img.
Thanks for any clues what might be wrong, Stefan
Version Platform Description
19:25:38-240594 INFO Starting SD.Next
19:25:38-244377 INFO Logger: file="/home/sseifried/vladmandic-webui/sdnext.log" level=INFO size=50387 mode=append
19:25:38-245575 INFO Python 3.10.6 on Linux
19:25:38-282334 INFO Version: app=sd.next updated=2023-12-30 hash=ab7b78cc url=https://github.com/vladmandic/automatic.git/tree/master
19:25:38-587679 INFO Latest published version: 5fb290f443d5f38a5c9f6e6095aabeab8e3a991d 2024-01-13T13:47:29Z
19:25:38-605954 INFO Platform: arch=x86_64 cpu= system=Linux release=6.7.0-arch3-1 python=3.10.6
19:25:38-617660 INFO Intel OneAPI Toolkit detected
19:25:38-671633 WARNING Modified files: ['.python-version']
19:25:38-685402 INFO Extensions: disabled=['Lora', 'sd-webui-controlnet', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'stable-diffusion-webui-images-browser',
'sd-extension-rembg']
19:25:38-687120 INFO Extensions: enabled=['stable-diffusion-webui-rembg'] extensions-builtin
19:25:38-688469 INFO Extensions: enabled=[] extensions
19:25:38-706712 INFO Command line args: ['--use-ipex'] use_ipex=True
19:25:43-123635 INFO Load packages: torch=2.1.0a0+cxx11.abi diffusers=0.25.0 gradio=3.43.2
19:25:43-609095 INFO Engine: backend=Backend.ORIGINAL compute=ipex mode=no_grad device=xpu cross-optimization="Scaled-Dot-Product"
19:25:43-635771 INFO Device: device=Intel(R) Arc(TM) A750 Graphics n=1 ipex=2.1.10+xpu
Relevant log output
19:25:47-976624 INFO Local URL: http://127.0.0.1:7860/
19:25:47-977896 INFO Initializing middleware
19:25:48-101664 INFO Startup time: 9.38 { torch=3.54 gradio=0.84 libraries=2.91 extensions=0.13 face-restore=0.18 upscalers=0.48 ui-img2img=0.06 ui-extras=0.39 ui-settings=0.21 ui-extensions=0.13
launch=0.15 api=0.12 }
19:26:12-781513 INFO MOTD: N/A
19:26:15-524562 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0
19:26:44-732052 INFO Select: model="dreamshaper_8 [879db523c3]"
Loading model: /home/sseifried/vladmandic-webui/models/Stable-diffusion/dreamshaper_8.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
19:26:44-797873 INFO Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=False bf16=True
19:26:50-443233 INFO LDM: LatentDiffusion: mode=eps
19:26:50-444645 INFO LDM: DiffusionWrapper params=859.52M
19:26:50-445948 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sseifried/vladmandic-webui/models/Stable-diffusion/dreamshaper_8.safetensors" size=2034MB
19:26:52-675449 INFO Applied IPEX Optimize.
19:26:52-676857 INFO Cross-attention: optimization=Scaled-Dot-Product options=[]
19:26:52-786481 INFO Load embeddings: loaded=0 skipped=0 time=0.10
19:26:52-787759 INFO Model loaded in 8.05 { create=5.62 apply=0.50 vae=0.67 move=0.88 hijack=0.18 embeddings=0.11 }
19:26:53-041281 INFO Model load finished: {'ram': {'used': 7.08, 'total': 31.3}, 'gpu': {'used': 2.03, 'total': 7.94}, 'retries': 0, 'oom': 0} cached=0
/build/intel-pytorch-extension/csrc/gpu/aten/operators/Indexing.h:670: operator(): global id: [0,0,0], local id: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed
/build/intel-pytorch-extension/csrc/gpu/aten/operators/Indexing.h:670: operator(): global id: [0,0,0], local id: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed
LIBXSMM_VERSION: main_stable-1.17-3651 (25693763)
LIBXSMM_TARGET: hsw [Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz]
Registry and code: 13 MB
Command: /home/sseifried/vladmandic-webui/venv/bin/python3 -u launch.py --use-ipex
Uptime: 75.365912 s
Backend
Original
Branch
Master
Model
SD 1.5
Acknowledgements
- [X] I have read the above and searched for existing issues
- [X] I confirm that this is classified correctly and its not an extension issue
Upstream IPEX issue: https://github.com/intel/intel-extension-for-pytorch/issues/505
Reproduced in the Docker image: https://github.com/Disty0/docker-sdnext-ipex/
Works fine in native Arch Linux.
Diffusers backend works fine in the Docker image. Use the Diffusers backend until the fix is implemented in IPEX itself.
@Disty0 thanks for your effort I had this issue on arch native using original backend.
I tried to confirm your findings with your docker container, what are your settings? For myself I tried to inpaint on a completely white 512x512 png file and diffusers backend keeps crashing. I keep all my settings on default, except switching from original to diffusers backend
I already disabled controlnet extension because otherwise it wouldn't even try to start.
Launching ipexrun launch.py...
/python/venv/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/python/venv/lib/python3.10/site-packages/intel_extension_for_pytorch/launcher.py:102: UserWarning: Backend is not specified, it will automatically default to cpu.
warnings.warn(
2024-01-20 09:23:54,133 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'jemalloc' memory allocator.
2024-01-20 09:23:54,133 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'auto' => 'intel' OpenMP runtime.
2024-01-20 09:23:54,134 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'taskset' multi-task manager.
2024-01-20 09:23:54,134 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: Untouched preset environment variables are not displayed.
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so:/python/venv/lib/libiomp5.so
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: KMP_AFFINITY=granularity=fine,compact,1,0
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: KMP_BLOCKTIME=1
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: OMP_NUM_THREADS=4
2024-01-20 09:23:54,135 - intel_extension_for_pytorch.cpu.launch.launch - INFO - cmd: taskset -c 0-3 /python/venv/bin/python3 -u launch.py -f --use-ipex --listen
09:23:54-344104 INFO Starting SD.Next
09:23:54-347766 INFO Logger: file="/sdnext/sdnext.log" level=INFO size=164782 mode=append
09:23:54-348989 INFO Python 3.10.12 on Linux
09:23:54-377820 INFO Version: app=sd.next updated=2024-01-13 hash=5fb290f4 url=https://github.com/vladmandic/automatic.git/tree/master
09:23:54-565654 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.7.0-arch3-1 python=3.10.12
09:23:54-573631 INFO Intel OneAPI Toolkit detected
09:23:54-637763 INFO Extensions: disabled=[]
09:23:54-639562 INFO Extensions: enabled=['Lora', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'stable-diffusion-webui-images-browser', 'sd-extension-system-info',
'stable-diffusion-webui-rembg', 'sd-webui-controlnet'] extensions-builtin
09:23:54-642838 INFO Extensions: enabled=[] extensions
09:23:54-644883 INFO Startup: quick launch
09:23:54-646622 INFO Verifying requirements
09:23:54-665184 INFO Verifying packages
09:23:54-667280 INFO Extensions: disabled=[]
09:23:54-668146 INFO Extensions: enabled=['Lora', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'stable-diffusion-webui-images-browser', 'sd-extension-system-info',
'stable-diffusion-webui-rembg', 'sd-webui-controlnet'] extensions-builtin
09:23:54-669682 INFO Extensions: enabled=[] extensions
09:23:54-672170 INFO Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
09:23:54-673575 INFO Command line args: ['-f', '--use-ipex', '--listen'] listen=True f=True use_ipex=True
09:23:58-715301 INFO Load packages: torch=2.1.0a0+cxx11.abi diffusers=0.25.0 gradio=3.43.2
09:23:59-139799 INFO Engine: backend=Backend.DIFFUSERS compute=ipex mode=no_grad device=xpu cross-optimization="Scaled-Dot-Product"
09:23:59-170120 INFO Device: device=Intel(R) Arc(TM) A750 Graphics n=1 ipex=2.1.10+xpu
2024-01-20 09:23:59.687105: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-20 09:23:59.687139: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-20 09:23:59.687169: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-20 09:24:00.768020: I itex/core/wrapper/itex_cpu_wrapper.cc:70] Intel Extension for Tensorflow* AVX2 CPU backend is loaded.
2024-01-20 09:24:01.003427: I itex/core/wrapper/itex_gpu_wrapper.cc:35] Intel Extension for Tensorflow* GPU backend is loaded.
2024-01-20 09:24:01.040545: I itex/core/graph/xpu_graph.cc:130] Please enable TF_ENABLE_ONEDNN_OPTS.So can benefit from the optimization of the MklCPUAllocator memory allocation.
2024-01-20 09:24:01.056139: I itex/core/devices/gpu/itex_gpu_runtime.cc:129] Selected platform: Intel(R) Level-Zero
2024-01-20 09:24:01.056173: I itex/core/devices/gpu/itex_gpu_runtime.cc:154] number of sub-devices is zero, expose root device.
09:24:01-372151 INFO Available VAEs: path="models/VAE" items=0
09:24:01-373452 INFO Disabled extensions: ['sd-webui-controlnet']
09:24:01-375418 INFO Available models: path="models/Stable-diffusion" items=1 time=0.00
09:24:01-557847 INFO Extension: script='scripts/faceid.py' [2;36m09:24:01-556285[0m[2;36m [0m[1;31mERROR [0m FaceID: No module named [32m'insightface'[0m
09:24:02-487770 INFO Extension: script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py' Using sqlite file: extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3
09:24:02-668395 INFO Extensions time: 1.14 { Lora=0.37 sd-webui-agent-scheduler=0.50 stable-diffusion-webui-images-browser=0.18 }
09:24:02-727770 INFO Load UI theme: name="black-teal" style=Auto base=sdnext.css
09:24:04-237446 INFO Local URL: http://localhost:7860/
09:24:04-238687 INFO Initializing middleware
09:24:04-420810 INFO [AgentScheduler] Task queue is empty
09:24:04-421935 INFO [AgentScheduler] Registering APIs
09:24:04-555356 INFO Startup time: 9.86 { torch=3.14 gradio=0.86 libraries=2.66 extensions=1.14 face-restore=0.15 upscalers=0.05 ui-extra-networks=0.08 ui-img2img=0.06 ui-control=0.05
ui-settings=0.18 ui-extensions=0.65 ui-defaults=0.09 launch=0.26 api=0.10 app-started=0.21 }
09:27:44-800417 INFO Select: model="dreamshaper_8 [879db523c3]"
Loading model: /sdnext/models/Stable-diffusion/dreamshaper_8.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
09:27:44-878954 INFO Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=False bf16=True
09:27:44-881047 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/sdnext/models/Stable-diffusion/dreamshaper_8.safetensors" size=2034MB
09:27:47-904247 INFO IPEX Optimize: time=0.16
09:27:47-916525 INFO Load embeddings: loaded=0 skipped=0 time=0.00
09:27:48-223151 INFO Load model: time=3.11 { load=3.11 } native=512 {'ram': {'used': 6.58, 'total': 31.3}, 'gpu': {'used': 2.06, 'total': 7.94}, 'retries': 0, 'oom': 0}
LIBXSMM_VERSION: main_stable-1.17-3651 (25693763)
LIBXSMM_TARGET: hsw [Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz]
Registry and code: 13 MB
Command: /python/venv/bin/python3 -u launch.py -f --use-ipex --listen
Uptime: 234.754138 s
Reproduced it with UniPC Sampler + Original Backend on native Arch Linux too. Diffusers is still working fine. Try starting the webui like this:
DISABLE_IPEXRUN=1 ./webui.sh --use-ipex
Also try switching to dev branch and try with using the system OneAPI. Install OneAPI basekit with your package manager and activate it, then start the webui like this:
DISABLE_VENV_LIBS=1 ./webui.sh --use-ipex
My settings for the Docker image is a brand new install with default settings but using Euler a sampler.
@Disty0 Forgot to mention, that it works on Arch native. It just doesn't work in docker for some reason.
Closing this as it is a year old issue. Feel free to open a new issu if this is still happening on latest Pytorch.