Note RF100 is part of the CVinW challenge!

Roboflow 100 📸: A Rich, Multi-Domain Object Detection Benchmark

This repository implements the Roboflow 100 benchmark developed by Roboflow. It contains code to download the dataset and reproduce mAP values for YOLOv5 and YOLOv7 Fine-Tuning and GLIP Evaluation on 100 of Roboflow Universe datasets.

RF100 was sponsored with ❤️ by Intel

RF100 paper is on arxiv, if you used RF100 in your research, please consider citing us

@misc{2211.13523,
Author = {Floriana Ciaglia and Francesco Saverio Zuppichini and Paul Guerrie and Mark McQuade and Jacob Solawetz},
Title = {Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
Year = {2022},
Eprint = {arXiv:2211.13523},
}

RF100

RF100 contains the following datasets, carefully chosen from more than 90'000 datasets hosted on our universe hub. The datasets are splitted in 7 categories: Aerial, Videogames, Microscopic, Underwater, Documents, Electromagnetic and Real World.

Note some datasets failed, only on 88

🚨 Head over the Appendix at the bottom to see samples from each dataset

We provide a notebook to help you using RF100 with PyTorch

Getting Started

First, clone this repo and go inside it.

git clone https://github.com/roboflow-ai/roboflow-100-benchmark.git
cd roboflow-100-benchmark
git submodule update --init --recursive

You will need an API key. RF100 can be accessed with any key from Roboflow, head over our doc to learn how to get one.

Then, export the key to your current shell

export ROBOFLOW_API_KEY=<YOUR_API_KEY>

Note: The datasets are taken from datasets_links.txt, you can modify that file to add/remove datasets.

Docker

The easiest and faster way to download RF100 is using docker and our Dockerfile.

NOTE Be sure to do the post process steps after you installed docker, we will read and write to shared volumes so your user should also be in the docker group.

If you have an NVIDIA GPU, be sure to also install nvidia docker

Be sure to have the ROBOFLOW_API_KEY in your env, then run it (we are passing our user so you shouldn't have permission errors on rf100 folder)

docker run --rm -it \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v $(pwd)/rf100:/workspace/rf100 \
    -v /etc/group:/etc/group:ro \
    -u "$(id -u):$(id -g)" \
    roboflow/rf100-download

roboflow/rf100-download is hosted on our docker hub

Internally, RF100 will downloaded to /app/rf100. You can also specify the format with the -f flag, by default coco is used.

docker run --rm -it \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/rf100:/workspace/rf100 \
    -v /etc/group:/etc/group:ro \
    -u "$(id -u):$(id -g)" \
    roboflow/rf100-download -f yolov5

If you want to build the container and not use the image on the hub, run

docker build -t rf100-download -f Dockerfile.rf100.download .

Local Env

To download RF100 in your local enviroment (python >=3.6), you need to install roboflow

pip install roboflow

Then,

chmod 770 ./scripts/download_datasets.sh
./scripts/download_datasets.sh
./scripts/download_datasets.sh -f yolov5 $ change format
./scripts/download_datasets.sh -l <path_to_my_location> change download location

Formats

Supported formats are

• coco
• yolov5 (used by YOLOv7 as well)

Reproduce Results

We will use docker to ensure the same enviroment is used.

First, build the container

docker build -t rf100-benchmark -f Dockerfile.rf100.benchmark .

Then, follow the guide for each model.

All results are stored inside ./runs.

Warning The run must be uninterrupted otherwise the downloaded datasets may be unfinished or corrupted

Warning If you have permission denied errors, you can either remove runs with sudo rm -rf ./runs or give permissions to everybody sudo chmod -R +x ./runs.

Warning Due to poor code support, random errors and time, we weren't able to run all the models on all datasets. Only 88 datasets were successfully benchmarked on all four models

YOLOv5 Fine-Tuning

Note, we will map the current folder to the container file system to persist data

mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./yolov5-benchmark/train.sh

YOLOv7 Fine-Tuning

Note, we will map the current folder to the container file system to persist data

mkdir -p runs &&
nvidia-docker run --gpus all --rm -d --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./yolov7-benchmark/train.sh

YOLOv8 Fine-Tuning

Note, Doc coming soon but we have the results (was done internally by another team member)

Note The model failed on these datasets: bone-fracture-7fylg, brain-tumor-m2pbp, cable-damage, circuit-elements and printed-circuit-board

GLIP

Note The model failed on these datasets: apex-videogame, apples-fvpl5, hand-gestures-jps7z, road-traffic, wall-damage

mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./GLIP-benchmark/train.sh

Note We create a notebook to analysis why and where GLIP did bad and good

COCO evaluation

If you are interesting in only the test annotations for rf100, e.g. you are using pycocotools to compute maps, we have create a zip file with root/<dataset_name>/<test_annotations>.json. We hope this will make it easier for you to evaluate your models.

Appendix

YOLOs Comparison

Note 5 datasets failed on v8, so the comparison is only on 95. Check v8 for more info.

Note We only train one seed for each model, take these results with a grain of salt

The following box plots shows [email protected] for each YOLOs model on RF100

The following bar plots shows [email protected] for each YOLOs model on each RF100 category

Samples

dataset	category	samples
hand-gestures-jps7z	real world
smoke-uvylj	real world
wall-damage	real world
corrosion-bi3q3	real world
excavators-czvg9	real world
chess-pieces-mjzgj	real world
road-signs-6ih4y	real world
street-work	real world
construction-safety-gsnvb	real world
road-traffic	real world
washroom-rf1fa	real world
circuit-elements	real world
mask-wearing-608pr	real world
cables-nl42k	real world
soda-bottles	real world
truck-movement	real world
wine-labels	real world
digits-t2eg6	real world
vehicles-q0x2v	real world
peanuts-sd4kf	real world
printed-circuit-board	real world
pests-2xlvx	real world
cavity-rs0uf	real world
leaf-disease-nsdsr	real world
marbles	real world
pills-sxdht	real world
poker-cards-cxcvz	real world
number-ops	real world
insects-mytwu	real world
cotton-20xz5	real world
furniture-ngpea	real world
cable-damage	real world
animals-ij5d2	real world
coins-1apki	real world
apples-fvpl5	real world
people-in-paintings	real world
circuit-voltages	real world
uno-deck	real world
grass-weeds	real world
gauge-u2lwv	real world
sign-language-sokdr	real world
valentines-chocolate	real world
fish-market-ggjso	real world
lettuce-pallets	real world
shark-teeth-5atku	real world
bees-jt5in	real world
sedimentary-features-9eosf	real world
currency-v4f8j	real world
trail-camera	real world
cell-towers	real world
apex-videogame	videogames
farcry6-videogame	videogames
csgo-videogame	videogames
avatar-recognition-nuexe	videogames
halo-infinite-angel-videogame	videogames
team-fight-tactics	videogames
robomasters-285km	videogames
tweeter-posts	documents
tweeter-profile	documents
document-parts	documents
activity-diagrams-qdobr	documents
signatures-xc8up	documents
paper-parts	documents
tabular-data-wf9uh	documents
paragraphs-co84b	documents
underwater-pipes-4ng4t	underwater
aquarium-qlnqy	underwater
peixos-fish	underwater
underwater-objects-5v7p8	underwater
coral-lwptl	underwater
aerial-pool	aerial
secondary-chains	aerial
aerial-spheres	aerial
soccer-players-5fuqs	aerial
weed-crop-aerial	aerial
aerial-cows	aerial
cloud-types	aerial
stomata-cells	microscopic
bccd-ouzjz	microscopic
parasites-1s07h	microscopic
cells-uyemf	microscopic
4-fold-defect	microscopic
bacteria-ptywi	microscopic
cotton-plant-disease	microscopic
mitosis-gjs3g	microscopic
phages	microscopic
liver-disease	microscopic
asbestos	microscopic
thermal-dogs-and-people-x6ejw	electromagnetic
solar-panels-taxvb	electromagnetic
radio-signal	electromagnetic
thermal-cheetah-my4dp	electromagnetic
x-ray-rheumatology	electromagnetic
acl-x-ray	electromagnetic
abdomen-mri	electromagnetic
axial-mri	electromagnetic
gynecology-mri	electromagnetic
brain-tumor-m2pbp	electromagnetic
bone-fracture-7fylg	electromagnetic
flir-camera-objects	electromagnetic

Credits

We thank all the authors of the original datasets, below a table linking the Roboflow 100 dataset's name to its original counterpart