automatic
automatic copied to clipboard
Published
20 hours ago •
vladmandic
vladmandic
[Issue]: Intel Ipex - Native API Failed
trafficstars
Issue Description
Spent most of the day generating a batch of 20 images that came out perfect just prior to this, then went to run a batch of 5 with a bit more complex prompt and face it crashed right at the end of the first picture, after face restore zoomed in on a face on the UI.
Repeatedly clicked generate again and again afterward and the button switches to "Starting" for a second then back to "Generate" without going to txt2img. Changing the prompt has no effect, but I have included extra crashes in the logfile incase they help to understand why.
Restarting web-ui resumes normal behaviour, although I'll have to update in the morning to tell you if it's having the same crash near the end of generation.
I have enclosed a log below, before the restart of web-ui.bat. Note that I have replaced the prompt with the word NSFW. It's a clean install with SDXL hardlinked from SwarmUI and optimized for ipex from the menu option, and SD2.1 from Gallery on it.
Version Platform Description
Web-UI is on latest Firefox. Arc Driver is 5590. Rest is below.
4:09:26-124526 INFO Starting SD.Next 04:09:26-133042 INFO Logger: file="C:\SD.Next\automatic\sdnext.log" level=DEBUG size=65 mode=create Python version=3.11.7 platform=Windows 11 04:09:26-340845 INFO Version: app=sd.next updated=2024-06-24 hash=94f6f0db branch=master url=https://github.com/vladmandic/automatic/tree/master ui=main 04:09:26-568403 DEBUG Branch sync failed: sdnext=master ui=main 04:09:27-647275 INFO Latest published version: 081c19fc122c6c8e60fcddfc37917e8107f65290 2024-07-01T08:20:42Z 04:09:27-657380 INFO Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD system=Windows release=Windows-10-10.0.22631-SP0 python=3.11.7 04:09:27-658380 DEBUG Setting environment tuning 04:09:27-659380 INFO HF cache folder: C:\Users\master.cache\huggingface\hub 04:09:27-660380 DEBUG Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512" 04:09:27-661381 DEBUG Torch overrides: cuda=False rocm=False ipex=True diml=False openvino=False 04:09:27-662384 DEBUG Torch allowed: cuda=False rocm=False ipex=True diml=False openvino=False 04:09:27-663384 INFO Python version=3.11.7 platform=Windows bin="C:\SD.Next\automatic\venv\Scripts\python.exe" venv="C:\SD.Next\automatic\venv" 04:09:27-667553 INFO Intel OneAPI Toolkit detected
Relevant log output
03:18:36-298511 DEBUG GC: utilization={'gpu': 137, 'ram': 50, 'threshold': 80} gc={'collected': 239, 'saved': 0.0}
beofre={'gpu': 10.58, 'ram': 8.0} after={'gpu': 10.58, 'ram': 8.01, 'retries': 0, 'oom': 0}
device=xpu fn=move_model time=0.38
03:19:59-524530 ERROR Exception: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI
error)
03:19:59-697778 ERROR Arguments: args=('task(1b2dpsfp2i9fvt0)', '**_NSFW_**', '', ['built-in\\cyberpunk style'],
99, 0, 26, True, True, False, False, 5, 1, 6, 6, 0.7, 0, 0.5, 1, 1, -1.0, -1.0, 0, 0, 0, 1024,
1024, False, 0.3, 2, 'None', False, 20, 0, 0, 10, 0, '', '', 0, 0, 0, 0, False, 4, 0.95, False,
0.6, 1, '#000000', 0, [], 0, 1, 'None', 'None', 'None', 'None', 0.5, 0.5, 0.5, 0.5, None, None,
None, None, 0, 0, 0, 0, 1, 1, 1, 1, None, None, None, None, False, '', 'None', 16, 'None', 1,
True, 'None', 2, True, 1, 0, True, 'none', 3, 4, 0.25, 0.25, 3, 1, 1, 0.8, 8, 64, True, True,
0.5, 600.0, 1.0, 1, 1, 0.5, 0.5, 'OpenGVLab/InternVL-14B-224px', False, False, 'positive',
'comma', 0, False, False, '', 'None', '', 1, '', 'None', 1, True, 10, 'None', True, 0, 'None',
2, True, 1, 0, 0, '', [], 0, '', [], 0, '', [], False, True, False, False, False, False, 0,
'None', [], 'FaceID Base', True, True, 1, 1, 1, 0.5, False, 'person', 1, 0.5, True) kwargs={}
03:19:59-747860 ERROR gradio call: RuntimeError
03:19:59-933291 DEBUG Server: alive=True jobs=1 requests=25145 uptime=52254 memory=1.64/15.93
backend=Backend.DIFFUSERS state=idle
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ C:\SD.Next\automatic\modules\call_queue.py:31 in f │
│ │
│ 30 │ │ │ try: │
│ ❱ 31 │ │ │ │ res = func(*args, **kwargs) │
│ 32 │ │ │ │ progress.record_results(id_task, res) │
│ │
│ C:\SD.Next\automatic\modules\txt2img.py:91 in txt2img │
│ │
│ 90 │ if processed is None: │
│ ❱ 91 │ │ processed = processing.process_images(p) │
│ 92 │ p.close() │
│ │
│ C:\SD.Next\automatic\modules\processing.py:191 in process_images │
│ │
│ 190 │ │ │ with context_hypertile_vae(p), context_hypertile_unet(p): │
│ ❱ 191 │ │ │ │ processed = process_images_inner(p) │
│ 192 │
│ │
│ C:\SD.Next\automatic\modules\processing.py:347 in process_images_inner │
│ │
│ 346 │ │ │ │ │ p.ops.append('face') │
│ ❱ 347 │ │ │ │ │ x_sample = face_restoration.restore_faces(x_sample, p) │
│ 348 │ │ │ │ │ if x_sample is not None: │
│ │
│ C:\SD.Next\automatic\modules\face_restoration.py:17 in restore_faces │
│ │
│ 16 │ face_restorer = face_restorers[0] │
│ ❱ 17 │ return face_restorer.restore(np_image, p) │
│ 18 │
│ │
│ ... 17 frames hidden ... │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl │
│ │
│ 1517 │ │ else: │
│ ❱ 1518 │ │ │ return self._call_impl(*args, **kwargs) │
│ 1519 │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl │
│ │
│ 1526 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1527 │ │ │ return forward_call(*args, **kwargs) │
│ 1528 │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\normalization.py:279 in forward │
│ │
│ 278 │ def forward(self, input: Tensor) -> Tensor: │
│ ❱ 279 │ │ return F.group_norm( │
│ 280 │ │ │ input, self.num_groups, self.weight, self.bias, self.eps) │
│ │
│ C:\SD.Next\automatic\modules\intel\ipex\hijacks.py:117 in functional_group_norm │
│ │
│ 116 │ │ bias.data = bias.data.to(dtype=weight.data.dtype) │
│ ❱ 117 │ return original_functional_group_norm(input, num_groups, weight=weight, bias=bias, eps=eps) │
│ 118 │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\functional.py:2558 in group_norm │
│ │
│ 2557 │ _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:])) │
│ ❱ 2558 │ return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled) │
│ 2559 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)
03:20:07-132869 DEBUG GC: utilization={'gpu': 188, 'ram': 10, 'threshold': 80} gc={'collected': 385, 'saved': 14.2}
beofre={'gpu': 14.58, 'ram': 1.65} after={'gpu': 0.38, 'ram': 0.42, 'retries': 0, 'oom': 0}
device=xpu fn=end time=5.91
03:22:00-171950 DEBUG Server: alive=True jobs=1 requests=25172 uptime=52374 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:24:00-217946 DEBUG Server: alive=True jobs=1 requests=25196 uptime=52494 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:26:00-267884 DEBUG Server: alive=True jobs=1 requests=25220 uptime=52614 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:28:00-315438 DEBUG Server: alive=True jobs=1 requests=25244 uptime=52734 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:30:00-365147 DEBUG Server: alive=True jobs=1 requests=25268 uptime=52854 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:32:00-425176 DEBUG Server: alive=True jobs=1 requests=25292 uptime=52974 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:34:00-467365 DEBUG Server: alive=True jobs=1 requests=25315 uptime=53094 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:35:59-511247 DEBUG Server: alive=True jobs=1 requests=25339 uptime=53213 memory=0.42/15.93
backend=Backend.DIFFUSERS state=idle
03:36:45-334060 DEBUG serialize sampler index: 0 as Default
03:36:45-457731 INFO [AgentScheduler] Total pending tasks: 1
03:36:45-472192 INFO [AgentScheduler] Executing task task(904kyydlr8sto1s)
03:36:45-477354 DEBUG Applying style: name="built-in\cyberpunk style" extra=[] skipped=[] reference=True
03:36:45-488399 DEBUG Pipeline class change: original=StableDiffusionXLInpaintPipeline
target=StableDiffusionXLPipeline device=cpu fn=init
03:36:45-523972 INFO Base: class=StableDiffusionXLPipeline
03:36:46-495946 ERROR Prompt parser encode: Native API failed. Native API returns: -999 (Unknown PI error) -999
(Unknown PI error)
03:36:46-502138 DEBUG Torch generator: device=xpu seeds=[3949873739]
03:36:46-505819 DEBUG Diffuser pipeline: StableDiffusionXLPipeline task=DiffusersTaskType.TEXT_2_IMAGE batch=1/20x1
set={'prompt': 1, 'negative_prompt': 1, 'guidance_scale': 7, 'num_inference_steps': 99, 'eta':
1.0, 'guidance_rescale': 0.7, 'denoising_end': None, 'output_type': 'latent', 'width': 1024,
'height': 1024, 'parser': 'Fixed attention'}
03:36:46-527858 ERROR Processing: args={'prompt': ['**_NSFW_** '], 'negative_prompt': [''],
'guidance_scale': 7, 'generator': [<torch._C.Generator object at 0x000001A74EB81110>],
'callback_on_step_end': <function diffusers_callback at 0x000001A74E96EFC0>,
'callback_on_step_end_tensor_inputs': ['latents', 'prompt_embeds', 'negative_prompt_embeds',
'add_text_embeds', 'add_time_ids', 'negative_pooled_prompt_embeds', 'negative_add_time_ids'],
'num_inference_steps': 99, 'eta': 1.0, 'guidance_rescale': 0.7, 'denoising_end': None,
'output_type': 'latent', 'width': 1024, 'height': 1024} Native API failed. Native API returns:
-999 (Unknown PI error) -999 (Unknown PI error)
03:36:46-531370 ERROR Processing: RuntimeError
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ C:\SD.Next\automatic\modules\processing_diffusers.py:122 in process_diffusers │
│ │
│ 121 │ │ else: │
│ ❱ 122 │ │ │ output = shared.sd_model(**base_args) │
│ 123 │ │ if isinstance(output, dict): │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\utils\_contextlib.py:115 in decorate_context │
│ │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py: │
│ │
│ 1086 │ │ │ negative_pooled_prompt_embeds, │
│ ❱ 1087 │ │ ) = self.encode_prompt( │
│ 1088 │ │ │ prompt=prompt, │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py: │
│ │
│ 398 │ │ │ │ │
│ ❱ 399 │ │ │ │ prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True) │
│ 400 │
│ │
│ C:\SD.Next\automatic\modules\intel\ipex\hijacks.py:186 in Tensor_to │
│ │
│ 185 │ else: │
│ ❱ 186 │ │ return original_Tensor_to(self, device, *args, **kwargs) │
│ 187 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)
03:36:46-762051 DEBUG Process interrupted: 2/20
03:36:46-764049 INFO Processed: images=0 time=1.28 its=0.00 memory={'ram': {'used': 0.58, 'total': 15.93}, 'gpu':
{'used': 0.53, 'total': 7.75}, 'retries': 0, 'oom': 0}
03:36:46-837953 INFO [AgentScheduler] Task queue is empty
03:37:09-267329 DEBUG serialize sampler index: 0 as Default
03:37:09-278647 INFO [AgentScheduler] Total pending tasks: 1
03:37:09-286649 INFO [AgentScheduler] Executing task task(v5d62zjdhxf3yyb)
03:37:09-289160 DEBUG Applying style: name="built-in\cyberpunk style" extra=[] skipped=[] reference=True
03:37:09-310681 INFO Base: class=StableDiffusionXLPipeline
03:37:09-327194 ERROR Prompt parser encode: Native API failed. Native API returns: -999 (Unknown PI error) -999
(Unknown PI error)
03:37:09-328698 DEBUG Torch generator: device=xpu seeds=[848539749]
03:37:09-329707 DEBUG Diffuser pipeline: StableDiffusionXLPipeline task=DiffusersTaskType.TEXT_2_IMAGE batch=1/20x1
set={'prompt': 1, 'negative_prompt': 1, 'guidance_scale': 7, 'num_inference_steps': 99, 'eta':
1.0, 'guidance_rescale': 0.7, 'denoising_end': None, 'output_type': 'latent', 'width': 1024,
'height': 1024, 'parser': 'Fixed attention'}
03:37:09-346802 ERROR Processing: args={'prompt': ['cyberpunk style **_NSFW_**'], 'negative_prompt': [''],
'guidance_scale': 7, 'generator': [<torch._C.Generator object at 0x000001A74EB83DD0>],
'callback_on_step_end': <function diffusers_callback at 0x000001A74E96EFC0>,
'callback_on_step_end_tensor_inputs': ['latents', 'prompt_embeds', 'negative_prompt_embeds',
'add_text_embeds', 'add_time_ids', 'negative_pooled_prompt_embeds', 'negative_add_time_ids'],
'num_inference_steps': 99, 'eta': 1.0, 'guidance_rescale': 0.7, 'denoising_end': None,
'output_type': 'latent', 'width': 1024, 'height': 1024} Native API failed. Native API returns:
-999 (Unknown PI error) -999 (Unknown PI error)
03:37:09-349316 ERROR Processing: RuntimeError
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ C:\SD.Next\automatic\modules\processing_diffusers.py:122 in process_diffusers │
│ │
│ 121 │ │ else: │
│ ❱ 122 │ │ │ output = shared.sd_model(**base_args) │
│ 123 │ │ if isinstance(output, dict): │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\torch\utils\_contextlib.py:115 in decorate_context │
│ │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py: │
│ │
│ 1086 │ │ │ negative_pooled_prompt_embeds, │
│ ❱ 1087 │ │ ) = self.encode_prompt( │
│ 1088 │ │ │ prompt=prompt, │
│ │
│ C:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py: │
│ │
│ 398 │ │ │ │ │
│ ❱ 399 │ │ │ │ prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True) │
│ 400 │
│ │
│ C:\SD.Next\automatic\modules\intel\ipex\hijacks.py:186 in Tensor_to │
│ │
│ 185 │ else: │
│ ❱ 186 │ │ return original_Tensor_to(self, device, *args, **kwargs) │
│ 187 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)
03:37:09-475957 DEBUG Process interrupted: 2/20
03:37:09-477962 INFO Processed: images=0 time=0.18 its=0.00 memory={'ram': {'used': 0.58, 'total': 15.93}, 'gpu':
{'used': 0.53, 'total': 7.75}, 'retries': 0, 'oom': 0}
03:37:09-528610 INFO [AgentScheduler] Task queue is empty
03:37:59-575646 DEBUG Server: alive=True jobs=1 requests=25367 uptime=53333 memory=0.58/15.93
backend=Backend.DIFFUSERS state=idle
03:39:59-616536 DEBUG Server: alive=True jobs=1 requests=25391 uptime=53453 memory=0.58/15.93
backend=Backend.DIFFUSERS state=idle
Backend
Diffusers
Branch
Master
Model
SD-XL
Acknowledgements
- [X] I have read the above and searched for existing issues
- [X] I confirm that this is classified correctly and its not an extension issue
P.S I'm really new to both SD and bug posting, so if I did anything wrong let me know. I checked Discussions and found nothing. Also I encountered this once before about two days ago, but it ran fine until today's one. I've ruled out GPU issues AFAIK.
from what i can tell, you stumbled on a bug in https://github.com/intel/intel-extension-for-pytorch which gets triggered under specific loads.
maybe @Disty0 can provide some info on best recommended version, but other than that, this is an upstream item.
Closing this as the issue thread is idle for months. Feel free to open a new issue if this is still happening on latest Pytorch.