excessive memory usage in pred, gt accumulation of COCOMetric

[matt-deboer]

🐛 Bug

Describe the bug During accumulate of COCOMetric, a large amount of memory is consumed storing the full pred and gt records. In my particular case, there are 15~20 bboxes per image, and the image sizes are 1500x1200. In all, this ends up using more than the available 32G on my system, and validation is never able to complete before being OOM-killed.

To Reproduce Steps to reproduce the behavior:

1. Perform mmdet + mask_rcnn + fastai training with latest 0.11 icevision, with COCOMetric attached/enabled, and using a dataset with relatively large images (1500x1200 in my case).
2. Monitor peak memory usage during the validation phase

Expected behavior Memory usage should be much lower; only the necessary details should be stored for calculating the COCOMetric results.

It seems the call to coco conversion functions could happen on each set of (gt, pred) as they come in, storing the bare minimum information needed for the computation.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: ubuntu 20.04

Additional context Add any other context about the problem here.

[fstroth]

I think the large memory usage is due to this line:

https://github.com/airctic/icevision/blob/b9d010d6fb1964d2725d3879b616b463627b20c7/icevision/models/mmdet/common/mask/prediction.py#L122

We should switch to using * REL* or EncodedRLEs which should save memory.