Super slow training when predicting bounding box orientations

[florianlanger]

Thanks for the great work and the amazing code base! I was interested in using your code to predict oriented bounding boxes. However, when doing so the training speed decreases dramatically compared to predicting oriented bounding boxes. I am training on a single RTX 2080 with 11 GB and bs of 7. The training time for 1080 epochs on ScanNet for the axis aligned bounding boxes is 30 hours. However, when I attempt to train to predict oriented bounding boxes (pretending scannet boxes have an orientation) the estimated train time shoots up to between 40 and 80 days!!! making it unusable for me. The main increase in time seems to come from computing 3D bounding box overlap with generalized_box3d_iou_tensor() and increased time in backpropagating gradients because I assume the computational graph has gotten a lot more complex. Do these numbers seem reasonable to you or am I missing something ? Is there some way to speed up the 3D bounding box overlap calculation ? Thanks!