Regression tests may fail if OS or compiler are different

[mwetter]

Some regression tests fail as they produce slightly different results on Windows and Linux, and possibly also if different compilers are used.

This issue should

• analysis these cases,
• write respective test cases that should pass and should fail and include them in the BuildingsPy regression tests, and
• update the tolerance check.

Maybe adding a relative test in addition to an absolute test may help.

[Mathadon]

I still wonder why the compiler would influence the numeric results of a computer program, since in theory the exact same program should give the exact same results. (unless random numbers etc are used of course) So I can only conclude it's not the exact same program.

A possible reason is that the integration algorithms in dymola are implemented differently between linux and windows.. A possible check could therefore be to run conflicting tests with Euler integration since there is no margin for interpretation in this integration algorithm.

[mwetter]

The same program compiled with different compilers, or with different optimization flags, can give different results. See also http://stackoverflow.com/questions/3354191/numerical-precision-differences-between-vs6-and-vs2008-using-c

For example, if you write t = t1 + t2 + t3, a compiler may cause the computation t = (t1 + t2) + t3 or t = t1 + (t2 + t3). The results may differ due to truncation errors.

[Mathadon]

Good point, I did not think of roundoff errors.. Still, these errors are much smaller than what we're seeing in our models?

We could try to make a branch where Annex60_Fluid_HeatExchangers_Examples_Heater_T.txt is solved using Euler and a step size of 1 s?

[mwetter]

I suggest we first look at the .mat files produced by Linux and Windows, and then see why the problem occurs and what to do to avoid it. There are various examples where we exhibit this. Also, for the MSL, people must also have looked into that issue I would think.

[thorade]

I was not able to execute the regessiontests on Windows. Are there installation instructions besides http://simulationresearch.lbl.gov/modelica/buildingspy/install.html ? The error is something like Error: Did not find executable even if I point it to the correct path.

[mwetter]

If dymola is on your path, the tests should run. The place where dymola is called is https://github.com/lbl-srg/BuildingsPy/blob/master/buildingspy/development/regressiontest.py#L27 with cmd=dymola.

[thorade]

I had a typo in the path, now it works! https://github.com/UdK-VPT/BuildingSystems/wiki/Add-Dymola-to-Windows-Path https://raw.githubusercontent.com/wiki/UdK-VPT/BuildingSystems/images/Dymola_Path.png

[thorade]

Maybe using csv-compare could be a solution here? https://github.com/modelica-tools/csv-compare Or is that used already?

[mwetter]

@Mathadon We are looking into that but having to install mono to get CSV compare to run on Linux seems not very attractive. I am afraid it will increase the burden on users to contribute models including reference results. It would be nice to have a Python or C/C++ port rather than .net.

[Mathadon]

I suppose that comment was for @thorade :)

[mwetter]

Yes, it was meant for @thorade

[thorade]

By burden you mean users do not want to install such a large package? Because installation is as easy as running sudo apt-get install mono-complete.

[thorade]

Commenting on an old issue, because I was thinking about it again. As I understand it, the csv-compare algorithms puts a "tube" around the reference results and checks whether the new results are within the tube. As I believe such an algorithm should already exist in python, I asked a related question on SO: https://stackoverflow.com/questions/46316442/compare-two-time-series-simulation-results