Investigate performance for combined models

[evetion]

Performance drops significantly (i.e. hangs) when creating a combined model (without interconnecting them), even though the models all run individually.

[visr]

I added the write_intermediate_models option in https://github.com/Deltares/Ribasim-NL/commit/4613021b5d32e4d31d4189e02d290f5cb0f1d22f to be able to investigate scaling a bit more. My hypothesis was that LU decomposition (in our case sparse KLU) is the scaling bottleneck, and that we may need to support an iterative method with an appropriate preconditioner. See also these recommendations.

Here are a few manually curated datapoints:

lhm-01: hws+agv lhm-02: hws+agv+delfland lhm-03: hws+agv+delfland+hollandsedelta

Model	Nodes	Computation Time (minutes)	KLU	check_negative_storage	jac	compute_rhs
lhm-01	1896	2.977	10%	11%	14%	39%
lhm-02	2584	4.994	10%	8%	30%	30%
lhm-03	6364	26.276

lhm-02.html.cpuprofile

For lhm-02 the profile buffer overflowed, though I assume the ratios didn't change much over time. It seems that most time is spent in our code formulating flows and such. That part has already been optimized before. Perhaps we should focus on trying to do less timesteps.

@Huite floated the idea of a custom AbstractController with ImplicitEuler that only does a solvability check on the Newton convergence to decrease timesteps, but not based on the estimated model error. Most hydrological models only do the solvability check. We can verify how much this will affect the results.

• [x] @SouthEndMusic also suggests looking at parallelizing water_balance!. And this line should be cached, because it is quite slow: https://github.com/Deltares/Ribasim/blob/e4b9fbd702a3f029bbdcd47711aaa43327de966a/core/src/solve.jl#L614

[visr]

Closed by https://github.com/Deltares/Ribasim/pull/2685. Next step is on the schematization side: https://github.com/Deltares/Ribasim-NL/issues/413. After that we can reevaluate the state of performance.