face-mask-detection
face-mask-detection copied to clipboard
NVIDIA-AI-IOT
tfrecords issue (tlt-dataset-convert)
Hi @ak-nv I am having an issue with creating the tfrecords for training. I used data2kitti.py to generate the training dataset, resulted in the following log
Directory Already Exists
Directory Already Exists
Kaggle Dataset: Total Mask faces: 4154 and No-Mask faces:790
Total Mask Labelled:4154 and No-Mask Labelled:790
Directory Already Exists
Directory Already Exists
MAFA Dataset: Total Mask faces: 1846 and No-Mask faces:232
Total Mask Labelled:6000 and No-Mask Labelled:1022
Directory Already Exists
Directory Already Exists
FDDB Dataset: Mask Labelled:0 and No-Mask Labelled:2845
Total Mask Labelled:6000 and No-Mask Labelled:3867
WideFace: Total Mask Labelled:0 and No-Mask Labelled:2134
----------------------------
Final: Total Mask Labelled:6000
Total No-Mask Labelled:6001
----------------------------
Just to mention, there was an ubyte overflow warning in lines 51 and 81 of the maffa2kitti.py file. I added the bbox coordinates in an int() to avoid the overflow. Even before I amend the code I had problem with the tfrecords, so I don't think this change has caused the issue with the tfrecords generation. Hopefully, the change I made was more of a fix than creating and issue.
So, assuming that the dataset is created correctly, lets get to the issue now.
The boxes don't seem to have the correct corners coordinates! Following the Jupyter Notebook, when I run tlt-dataset-convert I end of with quite a bit data samples being ignored. I guess this was mentioned in #5. I have got the latest version of the code as well "f62edc0"
The logs from running the tlt-dataset-convert as follows:
Any suggestions as what is causing this issue and how to resolve it please? Thanks
Converting Tfrecords for kitti trainval dataset
2020-09-29 14:22:57.557167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2020-09-29 14:23:00,181 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2020-09-29 14:23:00,181 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Creating output directory /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/tfrecords/kitti_trainval
2020-09-29 14:23:00,195 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 2827 Val: 706
2020-09-29 14:23:00,195 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2020-09-29 14:23:00,199 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2020-09-29 14:23:00,199 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this object
2020-09-29 14:23:00,280 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/396_mask.txt.
Coordinates: x1 = 635, x2 = 330, y1: 185, y2: 508
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/141_mask.txt.
Coordinates: x1 = 565, x2 = 378, y1: 102, y2: 321
Skipping this object
2020-09-29 14:23:00,381 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
2020-09-29 14:23:00,490 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/420_mask.txt.
Coordinates: x1 = 639, x2 = 318, y1: 232, y2: 506
Skipping this object
2020-09-29 14:23:00,580 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 586, x2 = 468, y1: 228, y2: 353
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 759, x2 = 655, y1: 134, y2: 255
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 237, x2 = 127, y1: 282, y2: 416
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 154, x2 = 51, y1: 157, y2: 271
Skipping this object
Top left coordinate must be less than bottom right.Error in object 5 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt.
Coordinates: x1 = 929, x2 = 827, y1: 278, y2: 399
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/55_mask.txt.
Coordinates: x1 = 363, x2 = 348, y1: 68, y2: 82
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/55_mask.txt.
Coordinates: x1 = 566, x2 = 530, y1: 133, y2: 176
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/55_mask.txt.
Coordinates: x1 = 462, x2 = 420, y1: 127, y2: 172
Skipping this object
2020-09-29 14:23:00,648 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/72_mask.txt.
Coordinates: x1 = 152, x2 = 98, y1: 133, y2: 192
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/72_mask.txt.
Coordinates: x1 = 530, x2 = 468, y1: 131, y2: 205
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/72_mask.txt.
Coordinates: x1 = 451, x2 = 408, y1: 145, y2: 200
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/72_mask.txt.
Coordinates: x1 = 278, x2 = 224, y1: 170, y2: 231
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/72_mask.txt.
Coordinates: x1 = 793, x2 = 721, y1: 129, y2: 214
Skipping this object
2020-09-29 14:23:00,717 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/227_mask.txt.
Coordinates: x1 = 755, x2 = 535, y1: 139, y2: 408
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/154_mask.txt.
Coordinates: x1 = 766, x2 = 573, y1: 84, y2: 323
Skipping this object
2020-09-29 14:23:00,785 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
2020-09-29 14:23:00,864 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/85_mask.txt.
Coordinates: x1 = 862, x2 = 645, y1: 82, y2: 292
Skipping this object
2020-09-29 14:23:00,934 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/5_mask.txt.
Coordinates: x1 = 408, x2 = 283, y1: 151, y2: 311
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/5_mask.txt.
Coordinates: x1 = 742, x2 = 667, y1: 194, y2: 271
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/5_mask.txt.
Coordinates: x1 = 899, x2 = 791, y1: 273, y2: 389
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/5_mask.txt.
Coordinates: x1 = 770, x2 = 704, y1: 112, y2: 180
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/445_mask.txt.
Coordinates: x1 = 741, x2 = 451, y1: 103, y2: 392
Skipping this object
2020-09-29 14:23:01,004 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'mask': 958
b'no-mask': 713
2020-09-29 14:23:01,004 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/63_mask.txt.
Coordinates: x1 = 669, x2 = 440, y1: 131, y2: 354
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/59_mask.txt.
Coordinates: x1 = 664, x2 = 532, y1: 167, y2: 328
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/136_mask.txt.
Coordinates: x1 = 751, x2 = 537, y1: 77, y2: 307
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/117_mask.txt.
Coordinates: x1 = 640, x2 = 435, y1: 108, y2: 341
Skipping this object
2020-09-29 14:23:01,280 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/113_mask.txt.
Coordinates: x1 = 535, x2 = 384, y1: 164, y2: 355
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/346_mask.txt.
Coordinates: x1 = 573, x2 = 343, y1: 117, y2: 447
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/66_mask.txt.
Coordinates: x1 = 465, x2 = 236, y1: 20, y2: 304
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/428_mask.txt.
Coordinates: x1 = 583, x2 = 185, y1: 128, y2: 514
Skipping this object
2020-09-29 14:23:01,547 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/335_mask.txt.
Coordinates: x1 = 716, x2 = 316, y1: 100, y2: 279
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/68_mask.txt.
Coordinates: x1 = 848, x2 = 783, y1: 78, y2: 155
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/68_mask.txt.
Coordinates: x1 = 243, x2 = 159, y1: 85, y2: 178
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/68_mask.txt.
Coordinates: x1 = 636, x2 = 560, y1: 38, y2: 133
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/68_mask.txt.
Coordinates: x1 = 527, x2 = 462, y1: 93, y2: 162
Skipping this object
2020-09-29 14:23:01,810 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/76_mask.txt.
Coordinates: x1 = 543, x2 = 425, y1: 170, y2: 321
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/76_mask.txt.
Coordinates: x1 = 364, x2 = 202, y1: 95, y2: 250
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/230_mask.txt.
Coordinates: x1 = 357, x2 = 175, y1: 127, y2: 342
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/249_mask.txt.
Coordinates: x1 = 697, x2 = 342, y1: 66, y2: 233
Skipping this object
2020-09-29 14:23:02,073 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/57_mask.txt.
Coordinates: x1 = 604, x2 = 499, y1: 81, y2: 214
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/469_mask.txt.
Coordinates: x1 = 732, x2 = 226, y1: 116, y2: 504
Skipping this object
2020-09-29 14:23:02,348 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/56_mask.txt.
Coordinates: x1 = 357, x2 = 306, y1: 241, y2: 295
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/56_mask.txt.
Coordinates: x1 = 162, x2 = 100, y1: 261, y2: 310
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/56_mask.txt.
Coordinates: x1 = 487, x2 = 412, y1: 178, y2: 259
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/56_mask.txt.
Coordinates: x1 = 97, x2 = 69, y1: 217, y2: 244
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/331_mask.txt.
Coordinates: x1 = 833, x2 = 545, y1: 162, y2: 472
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/350_mask.txt.
Coordinates: x1 = 732, x2 = 525, y1: 83, y2: 264
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/208_mask.txt.
Coordinates: x1 = 655, x2 = 119, y1: 71, y2: 544
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/73_mask.txt.
Coordinates: x1 = 858, x2 = 640, y1: 136, y2: 354
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/73_mask.txt.
Coordinates: x1 = 418, x2 = 173, y1: 162, y2: 359
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/93_mask.txt.
Coordinates: x1 = 534, x2 = 464, y1: 129, y2: 194
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/93_mask.txt.
Coordinates: x1 = 654, x2 = 612, y1: 199, y2: 255
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/93_mask.txt.
Coordinates: x1 = 332, x2 = 264, y1: 158, y2: 235
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/325_mask.txt.
Coordinates: x1 = 729, x2 = 215, y1: 168, y2: 444
Skipping this object
2020-09-29 14:23:02,616 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/165_mask.txt.
Coordinates: x1 = 360, x2 = 249, y1: 194, y2: 325
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/165_mask.txt.
Coordinates: x1 = 714, x2 = 584, y1: 108, y2: 257
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 743, x2 = 685, y1: 192, y2: 260
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 194, x2 = 128, y1: 124, y2: 192
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 575, x2 = 511, y1: 161, y2: 225
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 435, x2 = 380, y1: 125, y2: 177
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 888, x2 = 830, y1: 154, y2: 218
Skipping this object
Top left coordinate must be less than bottom right.Error in object 5 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 223, x2 = 197, y1: 165, y2: 198
Skipping this object
Top left coordinate must be less than bottom right.Error in object 6 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/8_mask.txt.
Coordinates: x1 = 355, x2 = 289, y1: 124, y2: 182
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/83_mask.txt.
Coordinates: x1 = 813, x2 = 675, y1: 160, y2: 276
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/83_mask.txt.
Coordinates: x1 = 514, x2 = 459, y1: 191, y2: 253
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/83_mask.txt.
Coordinates: x1 = 270, x2 = 141, y1: 183, y2: 292
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/403_mask.txt.
Coordinates: x1 = 667, x2 = 506, y1: 181, y2: 341
Skipping this object
2020-09-29 14:23:02,884 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/7_mask.txt.
Coordinates: x1 = 773, x2 = 482, y1: 85, y2: 386
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/103_mask.txt.
Coordinates: x1 = 423, x2 = 216, y1: 122, y2: 324
Skipping this object
2020-09-29 14:23:03,148 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/140_mask.txt.
Coordinates: x1 = 683, x2 = 508, y1: 138, y2: 377
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/123_mask.txt.
Coordinates: x1 = 749, x2 = 516, y1: 169, y2: 411
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/418_mask.txt.
Coordinates: x1 = 726, x2 = 343, y1: 114, y2: 329
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/109_mask.txt.
Coordinates: x1 = 818, x2 = 586, y1: 139, y2: 413
Skipping this object
2020-09-29 14:23:03,417 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/226_mask.txt.
Coordinates: x1 = 907, x2 = 422, y1: 139, y2: 544
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/493_mask.txt.
Coordinates: x1 = 725, x2 = 385, y1: 71, y2: 371
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/74_mask.txt.
Coordinates: x1 = 574, x2 = 326, y1: 118, y2: 374
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/25_mask.txt.
Coordinates: x1 = 424, x2 = 218, y1: 219, y2: 420
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/25_mask.txt.
Coordinates: x1 = 778, x2 = 572, y1: 141, y2: 364
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/61_mask.txt.
Coordinates: x1 = 633, x2 = 441, y1: 183, y2: 338
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/58_mask.txt.
Coordinates: x1 = 509, x2 = 443, y1: 136, y2: 205
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/58_mask.txt.
Coordinates: x1 = 247, x2 = 169, y1: 45, y2: 123
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/58_mask.txt.
Coordinates: x1 = 814, x2 = 735, y1: 82, y2: 170
Skipping this object
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/82_mask.txt.
Coordinates: x1 = 884, x2 = 828, y1: 229, y2: 302
Skipping this object
Top left coordinate must be less than bottom right.Error in object 1 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/82_mask.txt.
Coordinates: x1 = 466, x2 = 288, y1: 113, y2: 359
Skipping this object
Top left coordinate must be less than bottom right.Error in object 2 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/82_mask.txt.
Coordinates: x1 = 676, x2 = 578, y1: 227, y2: 361
Skipping this object
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/82_mask.txt.
Coordinates: x1 = 574, x2 = 530, y1: 229, y2: 300
Skipping this object
Top left coordinate must be less than bottom right.Error in object 4 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/82_mask.txt.
Coordinates: x1 = 90, x2 = 62, y1: 216, y2: 252
Skipping this object
2020-09-29 14:23:03,691 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'mask': 4001
b'no-mask': 2955
2020-09-29 14:23:03,691 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2020-09-29 14:23:03,691 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b'mask': 4959
b'no-mask': 3668
2020-09-29 14:23:03,691 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
b'Mask': b'mask'
b'No-Mask': b'no-mask'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2020-09-29 14:23:03,691 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.
I have the same issue.
@doruksonmez could you please share your logs as well? At the moment, looking at the samples that are ignored (starting with 114_mask.txt) they all seem to be belonging to the Kaggle dataset. I believe I previously had the same issue with some images from the MAFA datase as well, but there is no sign of those in my logs anymore! I'm not even sure why, I changed a couple of lines of code mafa2kitti.py (as explained previously).
Also, looking at the 114_mask.xml and then looking at my initial comment logs of 114_mask.txt, it's not clear whats happening
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this object
Compare this to the content of the 114_mask.xml
<xmin>924</xmin>
<ymin>133</ymin>
<xmax>598</xmax>
<ymax>571</ymax>
Not sure if the kaggle image labels are not being processed correctly or the labels are incorrect!
Hi,
I also have the same line along with the other similar outputs.
Top left coordinate must be less than bottom right.Error in object 0 of label_file /root/kitti-outs/train/labels/114_mask.txt.
Coordinates: x1 = 739, x2 = 478, y1: 90, y2: 388
Skipping this object
Regarding to other thing you mentioned about the pixel points, it shouldn't be an issue since data being rescaled to (960,544). However, I have a strange issue with my training procedure. It doesn't train at all and I can't see why.
Matching predictions to ground truth, class 1/2.: 100%|█| 16537/16537 [00:00<00:00, 192885.94it/s]
Matching predictions to ground truth, class 2/2.: 100%|█| 7640/7640 [00:00<00:00, 188193.63it/s]
Epoch 1/20
=========================
Validation cost: 0.015327
Mean average_precision (in %): nan
class name average precision (in %)
------------ --------------------------
mask nan
no-mask nan
...
Epoch 11/20
=========================
Validation cost: -0.000008
Mean average_precision (in %): 0.0000
class name average precision (in %)
------------ --------------------------
mask 0
no-mask 0
Median Inference Time: 0.009497
Regarding to other thing you mentioned about the pixel points, it shouldn't be an issue since data being rescaled to (960,544). However, I have a strange issue with my training procedure. It doesn't train at all and I can't see why.
@doruksonmez you're absolutely correct about the scaling, I didn't consider the fact that images are rescaled.
So we still don't know why some of the Kaggle samples are ignored during the tlt-dataset-convert.
@doruksonmez . Just to clarify. You are having the problem with tlt-dataset-convert generating the tfrecords but still continuing the training, despite the issue? Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is? As with your issue with the training, I have no idea why your training is not working!
@hosseinzadeh88 Yes, I am using tlt-dataset-convert tool for tfrecords and it skips some of the data but 3533 of them are still there and usable. So I tried increasing the category-limit to increase the size of the data and retried to train with that. But no luck. I think that's what you are seeing with the different sample counts, I am not trying with a custom dataset.
Yeah, it is frustrating to see my model is not being trained. But I am still trying to figure out what is wrong.
@hosseinzadeh88 and @doruksonmez
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt. Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
This is warning is if dataset has X2 <X1 , we convert in KITTI format which needs xmin, ymin, xmax, ymax format. You can have simple script update in mafa2kitti.py have a simple check here for xmin < xmax to avoid it. I have not check it yet though.
Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is?
In this case, your classifier will be biased towards predicting class 1
As with your issue with the training, I have no idea why your training is not working!
Which dataset are you using? What are your hyper-params. Can you share your detectnet_v2_train_resnet18_kitti.txt
Some minor fixes to improve accuracy
I just realized that, in widerface dataset, faces are obscure sometimes, so I now limit number of faces in the image. This might improve accuracy for faces without mask.
@ak-nv thanks for your comment.
I am using the exact same datasets you are using. You can find my conversion_spec.txt and detectnet_v2_train_resnet18_kitti.txt below.
conversion_spec.txt:
kitti_config {
root_directory_path: "/workspace/dataset/kitti-dataset"
image_dir_name: "train/images"
label_dir_name: "train/labels"
image_extension: ".jpg"
partition_mode: "random"
num_partitions: 2
val_split: 20
num_shards: 10
}
image_directory_path: "/workspace/dataset/kitti-dataset"
detectnet_v2_train_resnet18_kitti.txt:
random_seed: 42
dataset_config {
data_sources {
tfrecords_path: "/workspace/dataset/tfrecords/*"
image_directory_path: "/workspace/dataset/kitti-dataset"
}
image_extension: "jpg"
target_class_mapping {
key: "Mask"
value: "mask"
}
target_class_mapping {
key: "No-Mask"
value: "no-mask"
}
validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: "Mask"
value {
clustering_config {
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: "No-Mask"
value {
clustering_config {
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
}
model_config {
pretrained_model_file: "/workspace/dataset/resnet18.hdf5"
num_layers: 18
freeze_blocks: 0
use_pooling: False
all_projections: True
use_batch_norm: true
activation {
activation_type: "relu"
}
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: "resnet"
}
evaluation_config {
validation_period_during_training: 10
first_validation_epoch: 1
minimum_detection_ground_truth_overlap {
key: "Mask"
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: "No-Mask"
value: 0.5
}
evaluation_box_config {
key: "Mask"
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: "No-Mask"
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: "Mask"
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: "cov"
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: "bbox"
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: "No-Mask"
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: "cov"
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: "bbox"
initial_weight: 10.0
weight_target: 1.0
}
}
enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 16
num_epochs: 20
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-06
max_learning_rate: 5e-04
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 10
}
bbox_rasterizer_config {
target_class_config {
key: "Mask"
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: "No-Mask"
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}
Output for the conversion:
root@321bd4274bec:~/face-mask-detection# python3.6 data2kitti.py --kaggle-dataset-path /workspace/dataset/dataset/KaggleMedicalMaskDataset --mafa-dataset-path /workspace/dataset/dataset/MAFADataset --fddb-dataset-path /workspace/dataset/dataset/FDDBDataset --widerface-dataset-path /workspace/dataset/dataset/WiderFaceDataset --kitti-base-path /workspace/dataset/kitti-dataset --category-limit 6000 --tlt-input-dims_width 960 --tlt-input-dims_height 544 --train
Kaggle Dataset: Total Mask faces: 4154 and No-Mask faces:790
Total Mask Labelled:4154 and No-Mask Labelled:790
Directory Already Exists
Directory Already Exists
/root/face-mask-detection/data_utils/mafa2kitti.py:51: RuntimeWarning: overflow encountered in ubyte_scalars
bbox = [_bbox_label[0], _bbox_label[1], _bbox_label[0]+_bbox_label[2], _bbox_label[1]+_bbox_label[3]]
MAFA Dataset: Total Mask faces: 1846 and No-Mask faces:232
Total Mask Labelled:6000 and No-Mask Labelled:1022
Directory Already Exists
Directory Already Exists
FDDB Dataset: Mask Labelled:0 and No-Mask Labelled:2845
Total Mask Labelled:6000 and No-Mask Labelled:3867
WideFace: Total Mask Labelled:0 and No-Mask Labelled:2134
----------------------------
Final: Total Mask Labelled:6000
Total No-Mask Labelled:6001
----------------------------
@doruksonmez could you please run this script on your training dataset. I tried this on my dataset after successfully (apparently) running the data2kitti.py code, and to my surprise, I DO NOT have 6000 faces labeled as masked nor have 6001 labels of No-Mask! data2kitti.py is showing an incorrect total in my opinion. Please run this, I've also attached the script for you (it's slightly different as I was trying to extract all the face as well using opencv). Just have to change the datasetDir variable to point to the train directory, inside which you have 2 folders, images and labels.
import glob
maskedCounter = 0
noMaskCounter = 0
undefCounter = 0
datasetDir = '/home/sal/Desktop/Data4Training/train' # Main directory that contains two folders named "images" and the "labels"
# getting all the label files from the label folder within the datasetDir
fileNames = glob.glob(datasetDir+'/labels/'+'*.txt')
print(f'Total of {len(fileNames)} files found in the labels directory.')
for fileName in fileNames:
name = fileName.split('/')[-1]
name = name.split('.')[0]
# read the image and the labels
labelFile = open(datasetDir + '/labels/' + name + '.txt')
lines = labelFile.readlines()
for line in lines:
classLabel = line.split()[0]
#extract the faces from the images and write them to a seperate file.
if classLabel == 'No-Mask':
noMaskCounter = noMaskCounter + 1
elif classLabel == 'Mask':
maskedCounter = maskedCounter + 1
else:
undefCounter = undefCounter + 1
print(f'Total "MASK" labels: {maskedCounter}\nTotal "No-Mask" labels: {noMaskCounter} \nTotal undefined labels: {undefCounter}')
I think something is happening during the dataset generation, apparently, the widerfaces2kitti.py does not write to the disk or something like this, still trying to figure it out. @doruksonmez if you don't have 6000 "mask" and 6000 "no mask" labels please let me know so we can open another issue and inform the author.
This is what I get when I run this script:
Total of 3533 files found in the labels directory.
Total "MASK" labels: 5050
Total "No-Mask" labels: 3676
Total undefined labels: 0
Try the latest update from the author, 7a9f2c00280e206339d0f054216570429be640f4 They have updated the wider2kitti.py file which was not writing the files to the disk and was not even raising an exception to notice! So much time spent on training the network without actually having the correct dataset (@ak-nv ) I hope this has resolved the issue.
@doruksonmez
In your detectnet_v2_train_resnet18_kitti.txt ; image_directory_path should be "/workspace/dataset/kitti-dataset/train"
I assume, you are copying resnet18.hdf5 to dataset folder.
Let me know if this helps.
@ak-nv
Thanks for the comment. I tried to train with your detectnet_v2_train_resnet18_kitti.txt file inside tlt_specs and it worked!
This is the unpruned model evaluation output:
Validation cost: 0.001529
Mean average_precision (in %): 87.2487
class name average precision (in %)
------------ --------------------------
mask 88.3731
no-mask 86.1242
And this is for the pruned one:
Validation cost: 0.001639
Mean average_precision (in %): 86.4396
class name average precision (in %)
------------ --------------------------
mask 88.9616
no-mask 83.9176
However, when I export the pruned model and run DeepStream, I get a strange output bboxes as follows:
Good to hear that accuracy is close to what we had got. To resolve above issue with deepstream follow steps
Let me know if this solves it. I have also mentioned it in ReadMe.
@hosseinzadeh88 and @doruksonmez
Top left coordinate must be less than bottom right.Error in object 3 of label_file /home/Desktop/NvidiaFaceMaskDetection/FaceMaskData/train/labels/142_mask.txt. Coordinates: x1 = 419, x2 = 323, y1: 119, y2: 239
This is warning is if dataset has X2 <X1 , we convert in KITTI format which needs xmin, ymin, xmax, ymax format. You can have simple script update in
mafa2kitti.pyhave a simple check here for xmin < xmax to avoid it. I have not check it yet though.Also, are you using your own database? It seems to me as if your class 1/2 has 16537 samples and your class 2/2 has 7640. Is that what it is?
In this case, your classifier will be biased towards predicting
class 1As with your issue with the training, I have no idea why your training is not working!
Which dataset are you using? What are your hyper-params. Can you share your
detectnet_v2_train_resnet18_kitti.txtSome minor fixes to improve accuracy
I just realized that, in widerface dataset, faces are obscure sometimes, so I now limit number of faces in the image. This might improve accuracy for faces without mask.
For the X2 <X1 problem, you could add in kaggle2kitti.py
(xmin, xmax) = (xmin, xmax) if xmax >xmin else (xmax, xmin) # some bounding boxes are stored in inverse order (ymin, ymax) = (ymin, ymax) if ymax >ymin else (ymax, ymin) # some bounding boxes are stored in inverse order
at line 74 and line 84(86).
Just use the dataset that I have shared on google drive. See my other comments etc, it has 4000 masked and 4000 non-masked