centermask2 icon indicating copy to clipboard operation
centermask2 copied to clipboard

Enable support for RLE encoded segmentations

Open mjkvaak opened this issue 4 years ago • 7 comments

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

mjkvaak avatar Jun 16 '20 08:06 mjkvaak

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I met the same problem as you. Have you solved it now?

Schneey avatar Jun 26 '20 16:06 Schneey

Unfortunately no: I had to switch from RLE to polygons. Here's a link, in case you need to make COCO style polygon annotations yourself out of segmentation masks: https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch. However, I should mention that when simplifying the polygons' geometries with shapely, instead of using preserve_topology = False mentioned in the blog, one should prefer preserve_topology = True . The point is that in the former case a Polygon shape may get split into multiple polygons, a Multipolygon after the simplification. Then, a Multipolygon doesn't have the method for extracting the border points and the script will crash.

mjkvaak avatar Jun 27 '20 11:06 mjkvaak

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I also want to use bitmasks; the polygon annotations are tricky when simplifying using shapely, and also occupy larger space. In order to use bitmasks, one needs to modify mask_head.py. Somewhere the class BitMasks should be used as mentioned in https://github.com/facebookresearch/detectron2/issues/200#issuecomment-548163850 . However, I could not figure it out. Any help would be appreciated. Thanks.

fshamsafar avatar Aug 11 '20 08:08 fshamsafar

Hello @miasch @mjkvaak @Schneey ,

I am facing the same problem. how did yall solve the problem?

I am trying to implement CenterMask2 https://github.com/youngwanLEE/centermask2 which makes use of Detectron2. My annotations are in COCO RLE format, my data registration is successful. I visualized the data as well using the detectron2 visualizer, also I have set cfg.INPUT.MASk_FORMAT= bitmask, but somehow this error is being raised while I try to train the network.

Can somebody help me out here with what could be done?

/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/data/detection_utils.py:414: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)
  torch.stack([torch.from_numpy(np.ascontiguousarray(x)) for x in masks])
/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/fcos/fcos_outputs.py:402: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  per_candidate_nonzeros = per_candidate_inds.nonzero()
Traceback (most recent call last):
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 291, in <module>
    launch(
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/launch.py", line 62, in launch
    main_func(*args)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 285, in main
    return trainer.train()
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 162, in train
    self.train_loop(self.start_iter, self.max_iter)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 151, in train_loop
    self.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 423, in run_step
    self._trainer.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 228, in run_step
    loss_dict = self.model(data)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 166, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

Paragjain10 avatar Jan 13 '21 14:01 Paragjain10

If i remember correctly, I had the exact same error. I didn't ever put a lot of effort to try to solve problem, but switched to using conventional polygon annotations instead (see above).

mjkvaak avatar Jan 13 '21 17:01 mjkvaak

Hello @mjkvaak, Unfortunately, the application I am working on is such that I cannot use polygons as a mask. I have to use Rle segmentation mask only. Is there any other solution to this problem?

Is it possible to train Cenetrmask2 with Rle segmentation annotations?

Paragjain10 avatar Jan 25 '21 09:01 Paragjain10

If MASKIOU_ON = True, the error will come up. If MASKIOU_ON = False, the error will not take place.

cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)

The MASKIOU only can be trained with polygons instead of RLE format, but my dataset only has RLE format in segmentation.

15024287710Jackson avatar Sep 17 '22 07:09 15024287710Jackson