BrokenPipeError: [Errno 32] Broken pipe arises half-way when training on 2 GPUs

[YesDargo]

trafficstars

Similar to coco.py, I modifies the shape.ipynb to train on my dataset and first stage training on heads completed but the EOFError arises half-way during the first epoch training on 4+. The info is as below, I can train it on single GPU without this error.

There are 2k images in total and config like below. Is there problem with step_per_epoch or validation_steps?

GPU_COUNT = 2 IMAGES_PER_GPU = 5 STEPS_PER_EPOCH = 200 VALIDATION_STEPS = 200

Epoch 42/120
133/200 [==================>...........] - ETA: 3:13 - loss: 0.2841 - rpn_class_loss: 0.0012 - rpn_bbox_loss: 0.0529 - mrcnn_class_loss: 0.0424 - mrcnn_bbox_loss: 0.0377 - mrcnn_mask_loss: 0.150

Traceback (most recent call last):

134/200 [===================>..........] - ETA: 3:12 - loss: 0.2848 - rpn_class_loss: 0.0012 - rpn_bbox_loss: 0.0531 - mrcnn_class_loss: 0.0426 - mrcnn_bbox_loss: 0.0376 - mrcnn_mask_loss: 0.1502
---------------------------------------------------------------------------
BrokenPipeError                           Traceback (most recent call last)
<ipython-input-10-63fa505ed006> in <module>()
      4             learning_rate=config.LEARNING_RATE / 10,
      5             epochs=120,
----> 6             layers="4+")

~/Mask_RCNN/model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers)
   2236             max_queue_size=100,
   2237             workers=workers,
-> 2238             use_multiprocessing=True,
   2239         )
   2240         self.epoch = max(self.epoch, epochs)

~/miniconda3/envs/tf3/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name +
     90                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

~/miniconda3/envs/tf3/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   2210                 batch_index = 0
   2211                 while steps_done < steps_per_epoch:
-> 2212                     generator_output = next(output_generator)
   2213 
   2214                     if not hasattr(generator_output, '__len__'):

~/miniconda3/envs/tf3/lib/python3.6/site-packages/keras/utils/data_utils.py in get(self)
    758         """
    759         while self.is_running():
--> 760             if not self.queue.empty():
    761                 success, value = self.queue.get()
    762                 # Rethrow any exceptions found in the queue

<string> in empty(self, *args, **kwds)

~/miniconda3/envs/tf3/lib/python3.6/multiprocessing/managers.py in _callmethod(self, methodname, args, kwds)
    754             conn = self._tls.connection
    755 
--> 756         conn.send((self._id, methodname, args, kwds))
    757         kind, result = conn.recv()
    758 

~/miniconda3/envs/tf3/lib/python3.6/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

~/miniconda3/envs/tf3/lib/python3.6/multiprocessing/connection.py in _send_bytes(self, buf)
    402             # Also note we want to avoid sending a 0-length buffer separately,
    403             # to avoid "broken pipe" errors if the other end closed the pipe.
--> 404             self._send(header + buf)
    405 
    406     def _recv_bytes(self, maxsize=None):

~/miniconda3/envs/tf3/lib/python3.6/multiprocessing/connection.py in _send(self, buf, write)
    366         remaining = len(buf)
    367         while True:
--> 368             n = write(self._handle, buf)
    369             remaining -= n
    370             if remaining == 0:

BrokenPipeError: [Errno 32] Broken pipe

[billybee]

Hey @Xyza1972 - can I ask how did you prepare you own data? what tool you use to create label and mask in the VOC format?

[YesDargo]

I am doing binary task and each image only has one instance. I think data preparation is fine since I can train them successfully using one GPU and it also can train some steps using 2 GPUs then broken pipe error occured.

The training is on server and connected via ssh.

[austinmw]

I'm getting this too! same error at the same exact point in training.

Used pycococreator for the data. Have some very high resolution images (up to 6k), 3+1 classes.

Training in a nvidia-docker on GCE with 4 GPUs (P100's), 8 CPU, 37GB ram.

Switching from 4 GPUs to 1 seems to fix the issue.

[orestis-z]

I think this issue is critical, can we put some effort to fix it?

[derelearnro]

Do you have any solutions to solve it? I got the same problem and found some solutions, and still have this error

File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

I also tried to setup the parameters as below but still this error: IMAGES_PER_GPU = 1 USE_MINI_MASK = False

This problems may be occurred when I increasing the datasets.

[Gan-Tu]

are there any resolution to this? getting the same error

[derelearnro]

Yes, I solved it. Please reduce you image_pers_gpu as much as you can. It works fine for me now.

[JiangHao3]

I am getting the same issue using multiple GPUs. I have tried setting my images_per_gpu to be 1, and use_mini_mask = False. None of them solves the BrokenPipe problem. I am using Keras 2.1.3, python 3.6 in anaconda on Ubuntu16.4, GeForce GTX graphics card with 8 CPUs.

[truongtd6285]

I am getting the same issue using multiple GPUs. I have tried setting my images_per_gpu to be 1, and use_mini_mask = False. None of them solves the BrokenPipe problem. I am using Keras 2.1.3, python 3.6 in anaconda on Ubuntu16.4, GeForce GTX graphics card with 8 CPUs.

You can try to train on a smaller dataset, or use BACKBONE = "resnet50" instead of BACKBONE = "resnet101" by default.

[shradhakarnawat]

I am facing the same issue, but I am using CPU to train.

[AloshkaD]

I have the same issue. I have Nvidia 1080tis and 2080ti. The problem only occurs when I use the 2080ti for training but same code works perfectly fine on 1080ti. I figured this has to do with libraries, cuda, and other compatibility stuff. For now I've abandoned using the 2080ti and will try to reinstall all libraries when I get a chance.

[murdockhou]

I am facing the same issue, but I am using CPU to train.

Me too. I use one GPU and using generator to create dataset, model.fit to train model. And after some epochs, raise thie error.

[SYLin117]

I have tried to set use_multiprocessing to False, and it works, but really slow. I was testing on ubuntu 20 with 1070ti and AMD 3700X.

[alexberian]

I get the same issue with just 1 GPU when using multiprocessing. It seems to only come up when my data generator is using too many compute resources. Things that help are:

1. reduce the batch size
2. reduce the number of workers
3. reduce the max queue size
4. keep any processing in the data generator simple.
5. Don't use multiprocessing (very slow)