kohya_ss icon indicating copy to clipboard operation
kohya_ss copied to clipboard

Training stuck at the message "Loading settings from" in the Colab environment.

Open Skystapper opened this issue 6 months ago • 2 comments

The Logs are as per below

06:24:36-282863 INFO Command executed.
2024-08-26 06:24:47.122637: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-26 06:24:47.122681: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-26 06:24:47.124173: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-08-26 06:24:48.205464: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-08-26 06:24:49 INFO Loading settings from

I would've marked the issue with the title of TensorRT issue but someone here been saying that it can be ignored because kohya uses pytorch for the training.

I have everything installed with their latest version but the training process gets stuck with the same result every time. I can see that TensorRT is available there, yet as soon as I click on the "Start Training" button I get hit with these cuDDN, cuFFT, cuBLAS and TensorRT errors. If anyone here is working with Kohya in Colab's environment, would you please guide me on how to fix this issue.

Skystapper avatar Aug 26 '24 06:08 Skystapper