Model Convergence and Losses are not Decreasing

[shree-exofield]

I am trying to train the ellipse-rcnn model on my data, but its not converging at all; not even on the FDDB and dataset_sample.h5 as well. The loss_ellipse and total_loss only fluctuates. Other losses also show similar pattern at lower scale values. Would you tell me how to solve this issue? thereby I will have perfect detection for my elliptical object.