Should we remove update_sources_after_reflux?

[maxpkatz]

update_sources_after_reflux re-evaluates all source terms after a reflux on all levels. The logic is that the distribution of matter on the grid has changed, so the new-time source terms we evaluated will not have reflected this.

Historically, this approach originates with Miniati and Colella (2007), Section 3.2, and was in the original Castro code (see section 6.2.3 of the Castro paper), but it only dealt with self-gravity and was written in a way that the source correction due to the reflux was added. This was later turned into the current approach where we subtract the full new-time source and then re-calculate it and add it after accounting for the reflux contribution, and was also generalized to include all source terms.

The discussion question is: should we keep this approach, or should we just leave the source terms alone after a reflux? The argument for keeping it is that the evolution is more accurate. In particular, the evolution is approximately what it would be in an ideal case without subcycling (because in a no-subcycling mode like what FLASH does, we could calculate the fluxes on all levels first, before calculating the new-time sources). This enhanced accuracy matters if significant amounts of material move through a refinement boundary. It is not clear to me how the presence of this term affects convergence though. The argument for removing it is that it generally adds code complexity and it's easy for us to make mistakes in the advance that break the correctness of the source term update (cf. #1020). It also means we have to store more memory, because now we're keeping the fluxes array alive on every level indefinitely.

[zingale]

to be clear, I think we are second order accurate with or without it. If it works well, then I am happy to have it in, but it does not work with the new SDC stuff, for instance.