Custom labeler definitions cannot serialize to annotation_definitions

[twarrendewit]

Am introducing an extended version of the KeypointLabeler. Tried to use RegisterAnnotationDefinition to note the new annotation in annotation_definitions, but it looks like the serializable labelers are hard-code in the switch statement in PerceptionJsonFactory.Convert:

public static JToken Convert(PerceptionEndpoint consumer, string id, AnnotationDefinition annotationDefinition)
{
    switch (annotationDefinition)
    {
        case BoundingBoxDefinition def:
            return JToken.FromObject(LabelConfigurationAnnotationDefinition.Convert(def, "json", def.spec), consumer.Serializer);
        case BoundingBox3DDefinition def:
            return JToken.FromObject(LabelConfigurationAnnotationDefinition.Convert(def, "json", def.spec), consumer.Serializer);
        case InstanceSegmentationDefinition def:
            return JToken.FromObject(LabelConfigurationAnnotationDefinition.Convert(def, "PNG", def.spec), consumer.Serializer);
        case SemanticSegmentationDefinition def:
            return JToken.FromObject(PerceptionSemanticSegmentationAnnotationDefinition.Convert(def, "PNG"), consumer.Serializer);
        case KeypointAnnotationDefinition kp:
            return JToken.FromObject(PerceptionKeypointAnnotationDefinition.Convert(consumer, kp), consumer.Serializer);
    }
    return null;
}

https://github.com/Unity-Technologies/com.unity.perception/blob/Release-0.11.2-preview.2/com.unity.perception/Runtime/GroundTruth/Consumers/PerceptionJsonFactory.cs

Then tried extending KeypointLabeler to add in my own entries and still be valid in the above switch; unfortunately, all the constructors for KeypointLabeler are marked internal, so I can't actually add new ones to a new subclass. Oddly, they're not protected like that for BoundingBox, but I would like to emulate Keypoint's functionality.

Can we open it up to custom labelers somehow? RegisterAnnotationDefinition loses its utility when the allowed annotations are both hard-coded and closed for extension.

Very much appreciate any help!

[aryan-mann]

Hey @twarrendewit, this should be fixed with the upcoming release! 🎉

For the moment, you could try one of these two options:

1. Instead of adding the latest Perception package release, clone the repository and add it locally to your project via UPM. This will allow you to modify Perception source code and add an extra case to the above switch.
2. Create a custom copy of PerceptionEndpoint (and all dependent classes/structs) and modify that copy.

[twarrendewit]

Awesome -- thank you! Any timeline on the upcoming release?

[aryan-mann]

Hey @twarrendewit ! The Perception team just released 🎉 Perception 1.0 🎉, a major update to the toolset! We reworked how the Perception format was written to disk which includes removing the hard-coding of these definitions. We also recommend checking out the new SOLO output format (more details here: Perception 1.0: Expanding the Open Source Toolbox for Synthetic Data)

There are a bunch of cool features to check out such as ray tracing, depth, a new output format SOLO, normals, and more! Here is the full list of changes: Perception 1.0 Changelog

Closing this for now, feel free to open it back up if there are any issues.