Retinanet-Tutorial icon indicating copy to clipboard operation
Retinanet-Tutorial copied to clipboard

Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.

Open lucysumi opened this issue 9 months ago • 2 comments

Hi, I've been really stuck for days trying to figure this problem but couldn't..

So the training stops after like 15-17 epochs (I'm trying to run 300 epochs for my work), and displays this error message: Epoch 14: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-07. 50/50 - 32s - loss: 2.0636 - regression_loss: 1.8629 - classification_loss: 0.2007 - mAP: 0.6838 - lr: 1.0000e-05 - 32s/epoch - 649ms/step 2024-05-01 12:26:14.969075: W tensorflow/core/kernels/data/generator_dataset_op.cc:108] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]] #7 and #9 doesn't help. I did try the --step suggestion you mentioned but still didn't do much. At the max, I was able to run additional 3-4 epochs. i tried --steps 100, 50 and 10 .

Please help? @jaspereb

lucysumi avatar May 01 '24 12:05 lucysumi