Magnitude Normalization Adaptive Loss

[zoemcc]

Implementing the Magnitude Normalization adaptive loss method proposed in this paper "Optimally weighted loss functions for solving PDEs with Neural Networks" paper by Remco van der Meer, Cornelis Oosterlee, Anastasia Borovykh. The paper comes with a github repo which should help guide implementation. There's a lot of theory in the paper that has good background knowledge but perfect understanding of the analysis in the paper isn't required to implement the algorithm. The algorithm should be implemented as a concrete subtype of AbstractAdaptiveLoss so that it fits within our pre-existing code gen infrastructure in the discretize_inner_functions function.

(i.e.)

struct MagnitudeNormalizationAdaptiveLoss <: AbstractAdaptiveLoss
...
end

This seems like a useful method to compare against our other provided adaptive losses.

The weights each PDE loss term by a magnitude term (1/M_I and 1/M_B) that normalizes the relative magnitude of the loss by the square of the sum of the absolute values of each of the terms in the PDE equation, averaged over the relevant domain. The magnitude approximation should be strictly larger than the loss term because there is no cancellation in the magnitude term.

I think this is a harder method to implement than some of the other adaptive loss methods, because it requires breaking apart the given PDE functions at the symbolic level to calculate the M_I and M_B terms, and then integrating that quantity over the domains (possibly using the same strategy as the one given for the main PDE problem or a different strategy). Their method was designed for linear PDEs and it's possible there could be a theoretical issue to extending it straightforwardly to the nonlinear PDE case by just summing the absolute value of all the toplevel-add terms in the nonlinear PDE. If we define a more robust way of analyzing the PDE equations and synthesizing computational functions based off of that it could be worth doing, or it could just be a one-off bespoke analysis and synthesis for just this method.