[FEATURE] Optimize compilation speed by reorganize headers

[TonyXiang8787]

Introduction

The calculation core is a header-only library in C++. In each build target of unit test, C API, validation test, etc, we have multiple translation units (.cpp files). The compilation speed is slow in both CI and local development machine. This is frustrating and slows down the development.

Proposal

We would like to re-organize the headers in the header only library to reduce the dependencies between the headers. Hopefully we can increase the compilation speed. Possible ways of improvement can be:

1. Remove most of the standard library include in the main file power_grid_model.hpp. Only keep the absolute common standard libraries (like cstddef). Include the needed standard library in individual header files where it is needed.
2. Forward declare classes in the main file power_grid_model.hpp, so that other headers can use the opaque type if the full type is not needed. This reduces the dependencies between the headers.
3. Use separate header file with type_traits types to specify the input, update, .. types of each component. In this way, when compiling meta-data (like serializer), we do not need to include all the component types, only the traiter types.
4. Make the MathSolver a generic template class with supported solvers, just like how the MainModelImpl is templated with the components
5. Move generic template class using declarations like using MainModel = MainModelImpl<...>; to separate header files, e.g. main_model.hpp and main_model_impl.hpp. The tester of dispatch functionality of the MainModel could then use a component-less instance.

[mgovers]

I also think that the fact that we do not use builder patterns increases this issue. that's why e.g. test_math_solver.cpp needs all solvers, even though not all solvers are needed for all tests

We can do something with the math solvers in the same way as we do for the AllComponents in the MainModelImpl.

I also would recommend putting the using declaration in a separate header file, such that the tests that do not care about e.g. the iterative linear solver does not need to know about that solver.

Making smaller test files should also help, because doctest spends a lot of compilation time just on uncollapsing e.g. nested subcases

[mgovers]

irrelevant for CI but relevant for local development: i also think that we can split up the build target of the unit tests into multiple smaller ones to prevent the need of compiling all tests even though you're only looking into the deepest one

[TonyXiang8787]

@mgovers please modify the main content with your suggested header reorganization.

[TonyXiang8787]

irrelevant for CI but relevant for local development: i also think that we can split up the build target of the unit tests into multiple smaller ones to prevent the need of compiling all tests even though you're only looking into the deepest one

This could be the next step if the compilation speed is still not satisfying after optimizing the header structure. My guess is that the target compilation speed should be improved (with incremental build) when the headers are decoupled a bit.

[Jerry-Jinfeng-Guo]

This might be very novice input, but aren't we making heavy use of templates? I think normally that is also not the easiest thing when it comes to compiling.

[mgovers]

This might be very novice input, but aren't we making heavy use of templates? I think normally that is also not the easiest thing when it comes to compiling.

yes but its also the thing that makes our code fast, so that's part of the trade-off. making heavier use of concepts should solve this partially, because it first does a check on the constraints before trying to compile everything. this takes up some time but not as much as trying to compile an implementation until compilation fails and then try the next implementation (as is the case for regular templates).

[TonyXiang8787]

@mgovers

Is this issue still relevant? After last discussion we might choose a different way of approach?

[mgovers]

Although this ticket has majorly been repurposed compared to the initial setup, I think @figueroa1395 's work on getting the #718 should close this ticket.