Annotate rigid objects in 2D image with standard 3D cube

[hnuzhy]

My actions before raising this issue

I have read and searched the official docs and past issues for the solution. No one had the same problem with me.

Expected Behaviour

I want to annotate the head orientation of people in 2D image with a standard 3D cube. Here, the head is a rigid object. A standard cube is defined as follows: three sides of any vertex are perpendicular to each other, and all twelve sides are equal in length, or in unit length.

After labeling, we could get the eight projected vertices of the cube in the two-dimensional coordinate system. If three Euler angles (pitch, yaw, roll) are used to represent the orientation of the head, these precise projection points can be converted into corresponding angles.

Current Behaviour

• Current cuboid annotation The current provided cuboid annotation function in CVAT is not suitable for rigid object.

  1. Firstly, it can not guarantee that the edges of each vertex of the labeled cuboid are perpendicular to each other.
  2. Secondly, the length, width and height of cuboids are not necessarily equal.
  3. Finally, the side face of the current cuboid is always vertical. They can't be rotated. This makes it lack a dimension. These conditions make cuboid can not be used to mark the head orientation. In addition, I also think that such a cuboid is not suitable for labeling cars, chairs and other rigid objects.

• Alternative choice: ployline As an alternative, I try to annotate three consecutive non planar edges of the cube by using the ployline label. In this way, four points of the three edges can be used to estimate the Euler angles. However, this alternative can only solve the third problem of cuboid label mentioned above, and the first and second problems have not been solved. What we actually get are the rotated cuboids.

Possible Solution

I have three suggestions or roadmaps for adding unit cube label in the new version of CVAT.

1. Improve cuboid The current cuboid is actually oblique. However, objects in the real world should be marked with regular cuboids which satisfy that three edges of each vertex are perpendicular. At the same time, we need to release the third dimension of cuboid and allow it to rotate freely. I don't know if it's easy to implement with TypeScript. Three.js and other open source packages may be used for reference.

2. Modify cuboid-3d As far as I know, recent versions of CVAT already support 3D point cloud annotation. So is it possible to transplant the 3D cuboid module to the 2D image annotation? I'm not very familiar with the content of point cloud annotation, so it's inconvenient for me to give my opinions.

3. Add cube If possible, consider adding a new cube label to the candidate label button on the left side of CVAT. Users could choose to add new 3D cube graphics. The cube instance supports rotation at any angle on three dimensions. The software will automatically record the final Euler angle when the shape of cube is fixed.

Here are two examples of 3D model interaction. The first is the rotation interaction of a 3D head model in mayavi. The interactive operation needs to rely on both mouse and keyboard. The second is to use the 3D image editing tool in Windows 10 to place and operate 3D models on 2D images. All you need to do is use the mouse.

Example 1

Example 2

Next steps

Looking forward to your reply. I will be willing to do whatever I can to advance this functional part.