Represent integral by a neural network

[killah-t-cell]

From the conversation here https://github.com/SciML/NeuralPDE.jl/issues/390#issuecomment-921892269

We should ammortize the integral cost. I.e., if we integrate from (0,x), then the next point is at (0,y), we really only need to solve the integral on (x,y), but we instead do the whole thing.

We should also represent the integral itself by a neural network? So if I(...) is approximated by a neural network, it would be cheap to evaluate it at new points in the domain. Thus it would become a two-stage thing in the loss function:

1. Train the integral neural network NN_I to be a good approximation to the integral we want
2. Evaluate the PINN loss

(1) could use a lot of transfer learning: at each step u(x,y,t) does not change very much, so NN_I would be a decent approximation for the new integral, meaning that we could use the previous weights as the initial condition for the next one, even further decreasing the cost of (1).

• [ ] Ammortize integral cost by only solving necessary intervals
• [ ] Represent integral with a neural network
• [ ] Make Integration cheaper by representing using transfer learning to evolve it at each step

[killah-t-cell]

I was thinking about how to implement the integral with a neural network and would love to hear some thoughts re: what is the right way to approach/structure this?

I could see us – for example – simply writing get_neural_integral() which functions a bit like get_numeric_integral(). Inside it, we could take the integral representation we already have:

# example symbolic representation of an integral
integral_f = (cord, var"##θ#332", phi, derivative, integral, u_, p)->begin
          begin
              let (x,) = (cord[[1], :],)
                  begin
                      cord1 = vcat(x)
                  end
                  u_(cord1, var"##θ#332", phi)
              end
          end
end

and turn it into an optimization problem. We would then optimize it there first, and only when we have the result we can start optimizing the PDE and Bcs. Is that conversion analogous to how we process a PDE and Bcs? 1. build loss function, 2. generate training sets, 3. get loss function, 4. optimize? Or does it need to look different in a way – if so, how?

Are there any worries about accuracy here vs. the numerical integral?

The other thing we could do is to add this flow to discretize. I think a lot of the logic would be repeated otherwise.

BTW, Kirill's MWE here https://github.com/SciML/NeuralPDE.jl/pull/409#issuecomment-949865960 is a great starting point for this.