[FR] Add support for exporting COCO boxes and keypoints together

[jaderxnet]

System information

• Colab in Chrome
• https://colab.research.google.com/drive/1Lc3r2cUWTRCBOUdNlrq0Duo9TbSMyIbl#scrollTo=ymCSBcR78ZFo&uniqifier=4

Commands to reproduce

As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue. Application steps can be described in the next section.

export_dir = "/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset"
list_label = ["keypoints","bbox"]  # for example
print(list_label)
print(type(list_label))
# Export the dataset
dataset.export(
    export_dir=export_dir,
    dataset_type=fo.types.COCODetectionDataset,
    label_field=list_label,
)

Describe the problem

I need to convert a Brace (Link) based dataset to COCO Dataset format(Detection and Keyipoints). I managed to do a test with FiftyOne on Google Colab loading Bounding Box(Detection) and Keypoints (Link). But when I try to export to COCO format, I can only export Detection or Keypoints, not both together like two annotations on same image. In the documentation it is possible that the label_fild parameter is a list(link), but when I sent label_fild as a list, I got this error. I saw the implementation and maybe the lebel_field as a list is not implemented. Did I do something wrong? Can anyone help me with this?

Code to reproduce issue

Google Colab Link

ERROR LOGs

['keypoints', 'bbox']
<class 'list'>
Directory '/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset' already exists; export will be merged with existing files
WARNING:fiftyone.core.collections:Directory '/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset' already exists; export will be merged with existing files
   0% |/----------------|   0/153 [45.2ms elapsed, ? remaining, ? samples/s] 
INFO:eta.core.utils:   0% |/----------------|   0/153 [45.2ms elapsed, ? remaining, ? samples/s] 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-23-d64fdd810875>](https://localhost:8080/#) in <cell line: 6>()
      4 print(type(list_label))
      5 # Export the dataset
----> 6 dataset.export(
      7     export_dir=export_dir,
      8     dataset_type=fo.types.COCODetectionDataset,

5 frames
[/usr/local/lib/python3.10/dist-packages/fiftyone/core/collections.py](https://localhost:8080/#) in export(self, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, overwrite, **kwargs)
   7948 
   7949         # Perform the export
-> 7950         _export(
   7951             self,
   7952             export_dir=export_dir,

/usr/local/lib/python3.10/dist-packages/fiftyone/core/collections.py in _export(sample_collection, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, overwrite, **kwargs)
  10418 
  10419     # Perform the export
> 10420     foud.export_samples(
  10421         sample_collection,
  10422         dataset_exporter=dataset_exporter,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in export_samples(samples, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, num_samples, **kwargs)
    354         )
    355 
--> 356     write_dataset(
    357         samples,
    358         sample_parser,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in write_dataset(samples, sample_parser, dataset_exporter, num_samples, sample_collection)
    414         (UnlabeledImageDatasetExporter, LabeledImageDatasetExporter),
    415     ):
--> 416         _write_image_dataset(
    417             dataset_exporter,
    418             samples,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in _write_image_dataset(dataset_exporter, samples, sample_parser, num_samples, sample_collection)
    894 
    895                     # Export sample
--> 896                     dataset_exporter.export_sample(
    897                         image_or_path, label, metadata=metadata
    898                     )

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/coco.py](https://localhost:8080/#) in export_sample(self, image_or_path, label, metadata)
    824             labels = label.keypoints
    825         else:
--> 826             raise ValueError(
    827                 "Unsupported label type %s. The supported types are %s"
    828                 % (type(label), self.label_cls)

ValueError: Unsupported label type <class 'dict'>. The supported types are (<class 'fiftyone.core.labels.Detections'>, <class 'fiftyone.core.labels.Polylines'>, <class 'fiftyone.core.labels.Keypoints'>)

What areas of FiftyOne does this bug affect?

• [ ] App: FiftyOne application issue
• [x] Core: Core Python library issue
• [ ] Server: FiftyOne server issue

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

• [ ] Yes. I can contribute a fix for this bug independently
• [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community
• [x] No. I cannot contribute a bug fix at this time

[brimoor]

Unfortunately FiftyOne's builtin COCO exporter does not currently support merging data from multiple label fields (eg one Detections and one Keypoints) into a single JSON file on-the-fly.

[jaderxnet]

Thanks a lot for the quick response. I suggest updating the documentation description to make this clear. As soon as possible, I will collaborate for this feature to be implemented because I am an admirer of your work.

[brimoor]

Thanks for the kind words!

I think the way we can support this is by adding an optional keypoints parameter to COCODetectionDatasetExporter that is similar to iscrowd that allows the user to specify that their Detection instances have a an attribute that contain Keypoint instances that they'd like to include in the export. This attribute can be parsed here: https://github.com/voxel51/fiftyone/blob/de89c6d45669bf31489b17a4dd3b6bb59b4e5757/fiftyone/utils/coco.py#L1263

Assuming Dataset is properly constructed, the final syntax would then be like this:

dataset.export(
    labels_path="/path/for/coco.json",
    dataset_type=fo.types.COCODetectionDataset,
    label_field="<detetections_field>",
    keypoints="<keypoints_attribute>",
)

The problem with storing Detection and Keypoint in separate fields and passing them via the label_field argument is that there is no official way to know that two objects correspond to each other. I imagine users would prefer to store the keypoints as an attribute of the bounding box to keep the data better organized (we should add support for rendering such data in the FiftyOne App, too!)

[jaderxnet]

In fact, I'm specifically talking about files like person_keypoints_val2017.json, which have a structure something like this:

Array de annotations with: --- id --- segmentations Array with N Segmentations --- num_keypoints related with Segmentation --- area related with Segmentation --- keypoints Array with 17 Keypoints and confidence in flat 17 x 3 = 51 like [x,y,c,x,y,c,...] --- bbox Detection --- imageId --- category_id --- iscrowd ...

and each image can have multiple annotations. So, it seems to be a very different pattern from the tool's pattern. See a part of the file person_keypoints_val2017.json.

"annotations": [{"segmentation": [[125.12,539.69,140.94,522.43,100.67,496.54,84.85,469.21,73.35,450.52,104.99,342.65,168.27,290.88,179.78,288,189.84,286.56,191.28,260.67,202.79,240.54,221.48,237.66,248.81,243.42,257.44,256.36,253.12,262.11,253.12,275.06,299.15,233.35,329.35,207.46,355.24,206.02,363.87,206.02,365.3,210.34,373.93,221.84,363.87,226.16,363.87,237.66,350.92,237.66,332.22,234.79,314.97,249.17,271.82,313.89,253.12,326.83,227.24,352.72,214.29,357.03,212.85,372.85,208.54,395.87,228.67,414.56,245.93,421.75,266.07,424.63,276.13,437.57,266.07,450.52,284.76,464.9,286.2,479.28,291.96,489.35,310.65,512.36,284.76,549.75,244.49,522.43,215.73,546.88,199.91,558.38,204.22,565.57,189.84,568.45,184.09,575.64,172.58,578.52,145.26,567.01,117.93,551.19,133.75,532.49]],"num_keypoints": 10,"area": 47803.27955,"iscrowd": 0,"keypoints": [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,142,309,1,177,320,2,191,398,2,237,317,2,233,426,2,306,233,2,92,452,2,123,468,2,0,0,0,251,469,2,0,0,0,162,551,2],"image_id": 425226,"bbox": [73.35,206.02,300.58,372.5],"category_id": 1,"id": 183126},

[rusmux]

Are there any updates on this? The documentation is highly misleading and forces to dig into source code. At least let update the documentation

[julled]

It would be also nice to export boxes and segmentations at the same time.

[smidm]

I'm using following to add bboxes to the exported COCO dataset with keypoints only. It's just a bbox inferred from the keypoints, but it works for me.

import numpy as np
import json

def fix_coco(
        input_json,
        output_json,
):
    def iterable_to_int(t):
        return [int(x) for x in t]

    with open(input_json, "r") as f:
        coco_keypoints = json.load(f)

    for annotation in coco_keypoints["annotations"]:
        keypoints = np.array(annotation["keypoints"]).reshape(-1, 3)[:, :2]
        tl = keypoints.min(axis=0)
        br = keypoints.max(axis=0)
        if 'bbox' not in annotation:
            annotation["bbox"] = iterable_to_int(tuple(tl) + tuple(br - tl))
        if 'area' not in annotation:
            annotation["area"] = int((br[0] - tl[0]) * (br[1] - tl[1]))

    with open(output_json, "w") as f:
        json.dump(coco_keypoints, f)