horde-worker-reGen icon indicating copy to clipboard operation
horde-worker-reGen copied to clipboard

Post-processor crashing makes the worker stall

Open db0 opened this issue 1 year ago • 0 comments

From the logs of cozmyc, I noticed some weird OOM errors about Post-processors even though there should be more than enough VRAM. I don't quite understand why since I thought the CPU uses the RAM.

(Cozmyc has a very old CPU, so this is probably relevant)

2024-01-16 09:43:59.153 | ERROR    | hordelib.comfy_horde:send_sync:666 - execution_error, {'prompt_id': 'b0749e50-0fc3-423e-b157-d72a8511b395', 'node_id': 'face_restore_with_model', 'node_type': 'FaceRestoreWithModel', 'executed': ['model_loader', 'image_loader'], 'exception_message': 'Unable to allocate 384. MiB for an array with shape (4096, 4096, 3) and data type float64', 'exception_type': 'numpy.core._exceptions._ArrayMemoryError', 'traceback': ['  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 154, in recursive_execute\n    output_data, output_ui = get_output_data(obj, input_data_all)\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 84, in get_output_data\n    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 77, in map_node_over_list\n    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\nodes\\facerestore\\__init__.py", line 180, in restore_face\n    restored_img = face_helper.paste_faces_to_input_image()\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\nodes\\facerestore\\facelib\\utils\\face_restoration_helper.py", line 527, in paste_faces_to_input_image\n    inv_soft_mask * pasted_face + (1 - inv_soft_mask) * upsample_img\n'],

We should look at our error crashing in the PP process to make it fail more gracefully and inform the user.

db0 avatar Jan 16 '24 15:01 db0