Unwanted behaviour when using white background once normal loss starts

[RobinRenggli]

I'm training on a set of images that have been preprocessed by applying a segmentation mask, i.e. the background is white.

Training works well up until iteration 7000, which is until the normal loss kicks in:

However, after I start applying the normal loss, I get unwanted gaussians, such as seen here:

Which eventually grow to fill the entire background:

Is there any way to fix this behavior with the current implementation?