System tests failing with results differences in develop

[MakisH]

This is the first cron job that failed for all OpenFOAM cases.

I have not looked into details (@Eder-K how can I see the commit used, again? I don't see anything in the system tests output), but maybe @DavidSCN you have an idea what could be the issue?

[davidscn]

Best guess would be #148 , but the CHT fail as well and I haven't touched these parts.

[davidscn]

I think I got the reason: #148 introduced a new base class including an additional make target. If I understand the workflow correctly, the systemtest compares the build log (which changed due to the updated target) and the diff cascades down, have a look at the test failure. So, the tests are somehow outdated . Still not sure why the OpenFOAM - OpenFOAM case succeeds.

[MakisH]

Good point, but it does not seem to be the case here. Here I see the result files differing:

Files differing               : ['Fluid-Mesh-Centers-Solid.dt1.vtk', 'Fluid-Mesh-Centers-Solid.dt10.vtk',

[davidscn]

Might be related though. The diff comparison for the result files prints no further diffs on the console. Or do you have any more detailed information about the differences?

[davidscn]

I just checked the OpenFOAM-adapter changes locally using the perpendicular-flap test case and deal.II. I ran the current develop state of the openfoam-adapter against 290a26cad7173bc4e55e81423004fc1ab82fbbda

• diff precice-Solid-watchpoint-Flap-Tip.log openfoam-develop-watchpoint.log (of both participants)
• diff precice-Solid-iterations.log openfoam-develop-iterations.log
• diff dealii-solution-49.vtk openfoam-develop-dealii-solution-49.vtk

There are no diffs. Anything else I should check?

[MakisH]

Could you maybe also compare to master?

Also helpful would be to check with the reference data in the systemtests repository.

[davidscn]

Could you maybe also compare to master?

hm could do that, but are you aware that 290a26c passed in travis?

[MakisH]

I am not sure to what extent our system tests are behaving reasonably at the moment, so I am just trying to guess what could help us find more clues. :) But definitely comparing what you already got to the systemtests reference results would be the next step (e.g. here https://github.com/precice/systemtests/tree/develop/tests/TestCompose_dealii-of/referenceOutput)

Note that we are now only comparing the preCICE exported meshes.

And now that I remember, we did change the precision of the written VTK files in preCICE v2.2.0 (see https://github.com/precice/precice/pull/942). So a comparison would show if this is the issue.

[davidscn]

Some notes:

• the test already failed in the respective float double update https://github.com/precice/systemtests/pull/275
• for the precice*iteration files, the reference is outdated due to a more comprehensive output by the current precice version. Reference has TimeWindow TotalIterations Iterations Convergence
• where can I find the actual config used by the tests? If we just pull the current state of the tutorials.. they changed quite a lot (in terms of testing) https://github.com/precice/tutorials/commits/master/FSI/flap_perp_2D/OpenFOAM-deal.II/precice-config.xml

[MakisH]

Thank you, @DavidSCN, good observations! I guess we need to update the reference results then, after checking first that all the differences are only in terms of format and precision.

The system tests directly use the configs of the tutorials, manually editing small things such as the simulated time.

[davidscn]

Coming to an end here: the precision of the reference vtk (17) data still differs from the one I generate with preCICE v2.2 (18):

9,87c9,87
< -5.0000000000000003e-02  0.0000000000000000e+00  0.0000000000000000e+00
< -5.0000000000000003e-02  5.5555555555555552e-02  0.0000000000000000e+00
< -5.0000000000000003e-02  2.7777777777777776e-02  0.0000000000000000e+00
< 5.0000000000000003e-02  0.0000000000000000e+00  0.0000000000000000e+00
< 5.0000000000000003e-02  5.5555555555555552e-02  0.0000000000000000e+00
---
> -5.00000000000000028e-02  0.00000000000000000e+00  0.00000000000000000e+00
> -5.00000000000000028e-02  5.55555555555555525e-02  0.00000000000000000e+00
> -5.00000000000000028e-02  2.77777777777777762e-02  0.00000000000000000e+00
> 4.99999999999999958e-02  0.00000000000000000e+00  0.00000000000000000e+00
> 4.99999999999999958e-02  5.55555555555555525e-02  0.00000000000000000e+00

For me it fails to reproduce the reference data and I think trying to do so is a waste of time. I have already compared 290a26c against the current develop state. If I don't use the 2D feature for now (since it is not available in master) and diff the coupling data of the current develop against the current master, I don't get any differences. Let us finish the topic here and update the references. (Off-topic) I would propose to reduce the number of reference files.

[MakisH]

For me it fails to reproduce the reference data and I think trying to do so is a waste of time.

It is clear now that we should not try to reproduce the previous reference results, but update the reference files to fit the results. We just need to understand this difference every time we see it, before updating the reference.

(Off-topic) I would propose to reduce the number of reference files.

We already reduced them by a very large amount (replaced solver results by preCICE results) and maybe indeed we can reduce them even further. Feel free to open an issue for that in the system tests, explaining this idea.