Splitting in X for FCI

[ZedThree]

I think it should be possible to split in X for FCI if we can be sure that there are sufficient X guard cells such that the field line end points can still be interpolated to. We might end up in a slightly strange situation where the interpolation needs to be done over a slightly larger region than the interior points but still not over the whole domain.

The downside here is that you probably end up needing 3 or 4 guard cells in X, which might not be great at scale due to increased computation and communication.

The FCI initialisation probably also assumes no splitting in X, so it might need adapting to use global X indexing.

[d7919]

At scale I think the communication cost probably won't mind too much if it's 1 guard cell or 4 as small ish messages are often ~constant cost. I think the issue of calculation in the guard cells will probably have a more significant impact on scaling efficiency. Things might be ok if we avoid operating in the guards too much by using appropriate regions, where possible.

[bendudson]

I think this is a good idea and will probably work in the vast majority of cases; where it doesn't work we can detect that and fail with a message. If the field lines are too scrambled up between y planes then the grid is insufficiently resolved. Another possibility is to treat FCI interpolation as a matrix-vector product, so we can use PETSc or similar to work out the communication pattern when we assemble the matrix. This has some advantages in that we can calculate the transpose of this matrix for use in creating divergence operators from gradient operators (as done in Grillix).