rework pycocotools with faster-coco-eval

[MiXaiLL76]

trafficstars

Hello, I suggest to include the fast validation library "faster-coco-eval" for validating coco & lvis datasets.

The guys from lightning.ai have been using it for a long time as an alternative backend for validation, instead of the dying pycocotools.

https://lightning.ai/docs/torchmetrics/stable/detection/mean_average_precision.html

I prepared examples of this library in advance for you, they are available here:

https://github.com/MiXaiLL76/faster_coco_eval/blob/main/examples/comparison/ultralytics/colab_example.ipynb or https://nbviewer.org/github/MiXaiLL76/faster_coco_eval/blob/main/examples/comparison/ultralytics/colab_example.ipynb

For quick comparison, here is a table from the end of this document:

eval compare considering data loading

lib	model	time	profit
faster-coco-eval	bbox	7.34	4.5x
pycocotools	bbox	34.35	1x
----	-----	----	------
faster-coco-eval	bbox+segm	26.51	3.2x
pycocotools	bbox+segm	86.79	1x
----	-----	----	------
faster-coco-eval	bbox+keypoints	4.45	2,2x
pycocotools	bbox+keypoints	10.17	1x

Additional information:

I recently helped integrate it into D-FINE and RT-DETR. It works great there.

Maybe this will help you not only train fast models, but also train them faster =)

I want to offer you an even easier way to integrate your developments!

I implemented and tested two modules:

FasterCocoDetection - a dataset that uses faster-coco-eval instead of pycocotools in torchvision https://github.com/MiXaiLL76/faster_coco_eval/blob/main/faster_coco_eval/utils/pytorch/coco_dataset.py
FasterCocoEvaluator - a validator that is a clone of DETR validators, configured to work quickly with faster-coco-eval https://github.com/MiXaiLL76/faster_coco_eval/blob/main/faster_coco_eval/utils/pytorch/coco_eval.py

Why did I do this? - I'm trying to tidy up a bunch of repositories that can use common developments.

For you I have implemented and tested the validation with the implementation of these functions, to check you need to study my code and update the library to version 1.6.6 (available on pypi & conda)

Less code - more time to think about great things)

Unfortunately, I can't fully test the solutions because I don't have the data and power to run them now, but I hope we can collaborate and implement my changes thanks to your tests)

[CLAassistant]

All committers have signed the CLA.

[Matvezy]

Hello @MiXaiLL76 thanks a lot for opening the PR! We didn't have a chance to look at it yet but will review it in the near future :)

[MiXaiLL76]

@Matvezy

Hi! Tell me, is there a simple way to run validation in your framework without training?

I don't have many resources to run training for checking, but I can run validation for testing

[MiXaiLL76]

@Matvezy Should I continue working on the project? Or will you stick with pycocotools?

The guys at Ultralytics have already implemented it. https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/yolo/detect/val.py#L462