ModelicaStandardLibrary icon indicating copy to clipboard operation
ModelicaStandardLibrary copied to clipboard
verified publisher icon modelica

MSL 4.1.0 Regressions - MultiBody

Open GallLeo opened this issue 1 year ago • 5 comments

The following models fail in result comparison. Tested revision: f9bddf86 (2024-02-16)

Changed models, need reference update after library officer check:

  • [x] Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2 - Reason: very tiny error at the end of the simulation on freeMotion.v_rel_a[2], which is very close to zero. The problem here is that CSV compare is just too strict on variables close to zero. One option is to remove the variable until we have a proper solution for that. This is done by PR #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421. - Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
  • [x] Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D - Reason: the trajectories diverge sharply for time > 0.52, most likely due to chaotic motion - the system is highly nonlinear and has more than two states, which are necessary ingredients for the path to chaos. Just changing Dymola version produces a different trajectory after that point, and also OpenModelica behaves differently. There's no point trying to reproduce chaotic trajectories, so we can limit the time span to StopTime = 0.4 only for the reference results. CSV compare should stop comparing when it no longer has data, but it needs to be fixed, see modelica-tools/csv-compare#65 - Updated reference files? @GallLeo please generate reference files with StopTime = 0.4.
  • [x] ModelicaTest.MultiBody.Parts.FreeBodies - Reason: body.Q[2] is very close to zero, same story as the first example. Interim solution: remove it from the reference trajectories. This could be done by PR #4420 (needs to be ported to maint/v4.1.0), however, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421. - Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
  • [x] ModelicaTest.MultiBody.Sensors.AbsoluteSensor - Reason: Test model changed, see https://github.com/modelica/ModelicaStandardLibrary/pull/4148 (and esp. https://github.com/modelica/ModelicaStandardLibrary/commit/82d1b167e20e0decf632a8c9a2ea58f3c92415cb). The new results are correct. - Updated reference files? @GallLeo please update reference files.

No signals to compare, define more comparison signals:

  • [X] ModelicaTest.MultiBody.Visualizers.Arrow - Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial - Updated reference files? No need to
  • [X] ModelicaTest.MultiBody.Visualizers.ColorMaps - Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial - Updated reference files? No need to
  • [X] ModelicaTest.MultiBody.Visualizers.Shapes - Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial - Updated reference files? No need to
  • [X] ModelicaTest.MultiBody.Visualizers.Torus - Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial - Updated reference files? No need to

Change due to new Dymola version

  • [x] ModelicaTest.MultiBody.Joints.Universal - Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. Francesco: I further analyzed the issue after the meeting with @tobolar. universal.w_a is zero until about time = 0.3, then it starts showing some oscillations which are growing over time, but are clearly triggered by numerical errors and have a negligible amplitude (2e-8 rad/s). This is also confirmed by looking at the regression in OpenModelica. There is no point trying to reproduce this signal, which clearly only depends on numerical approximations, so until CSV compare can handle that properly, we can remove it in PR #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421. - Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.

  • [x] ModelicaTest.MultiBody.Joints.UniversalSpherical - Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. We made further analysis, in this case there are several variables which are very close to zero, similarly to PointGravityWithPointMasses2. We should remove all variables that are close to zero, because there is currently no way to test them reliably with CSV-compare, and on the other hand it is unlikely that these are wrong while the other ones are OK, since they are components of some physical vectors. This can be done by PR #4420 (needs to be ported to maint/v4.1.0) However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421. - Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.

  • [x] ModelicaTest.MultiBody.Parts.Rotor1D.GearConstraint3 - Reason: chaotic motion with bifurcation after time = 0.30 - Updated reference files? @GallLeo please generate reference files with StopTime = 0.30. Also needs fixing modelica-tools/csv-compare/#65 to avoid a false positive

  • [X] ModelicaTest.MultiBody.PlanarLoopWithMove - Reason: No more regression, probably due to resolved table generator issues. - Updated reference files? No.

  • [X] ModelicaTest.MultiBody.Sensors.Distance2 - Reason: No more regression, probably due to resolved table generator issues. - Updated reference files? No

  • [X] ModelicaTest.MultiBody.Sensors.RelativeSensor - Reason: No more regression, probably due to resolved table generator issues. - Updated reference files? No.

  • [ ] Release notes check: Are all classes mentioned which could lead to result changes in user models?

Useful Links

Current comparison report: https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/comparison_report_overview.html -> Reference result test -> Comparison

Comparison signal definitions: https://github.com/modelica/ModelicaStandardLibrary/tree/master/Modelica/Resources/Reference/Modelica https://github.com/modelica/ModelicaStandardLibrary/tree/master/ModelicaTest/Resources/Reference/ModelicaTest

Reference results: https://github.com/modelica/MAP-LIB_ReferenceResults

GallLeo avatar Feb 27 '24 08:02 GallLeo

Library officer check:

  • Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2 Checking https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/20240227232653/Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2/CSVCompare/PointGravityWithPointMasses2_report.html, the error message is: 6 (relative error is 1,81520807176844E-16) So I really wonder why csv-compare catch this. Anyway - this example is fine to me.

  • Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D I simulated both MSL400 and MSL4100 in Dymola2023x Refresh1, and then MSL400 and MSL4100 in Dymola2024x. Comparing results of the two MSL versions calculated in ONE of Dymola yields the same results. So the problem seems to be the comparison across the Dymola versions. I tried to increase the numerical tolerance but this alone doesn't help. What seems could help is to a) introduce a damping and b) increase the tolerance (to=1e-6). So maybe better place this under "Change due to new Dymola version" below?

  • ModelicaTest.MultiBody.Parts.FreeBodies Checking https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/ModelicaTest/20240228011806/ModelicaTest.MultiBody.Parts.FreeBodies/CSVCompare/FreeBodies_report.html, the error message is: 433 (relative error is 3,22845694576259E-13) So I really wonder why csv-compare catch this. Anyway - this example is fine to me.

  • ModelicaTest.MultiBody.Sensors.AbsoluteSensor Test model changed, see https://github.com/modelica/ModelicaStandardLibrary/pull/4148 (and esp. 82d1b167e20e0decf632a8c9a2ea58f3c92415cb) So the reference files shall be updated.

No signals to compare, define more comparison signals:

  • ModelicaTest.MultiBody.Visualizers The tests are intended to test only 3d visualization of the objects. So not sure how to proceed here.

Change due to new Dymola version (reference update after library officer check?):

  • ModelicaTest.MultiBody.Parts.Rotor1D.GearConstraint3 This seems to be similar to Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D above. So a) introduce a damping and b) increase the tolerance (to=1e-6).

Not classified, yet (needs library officer check):

  • Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D This item is doubled, see the same class above.

  • ModelicaTest.MultiBody.PlanarLoopWithMove I can't simulate - maybe missing binaries? https://github.com/modelica/ModelicaStandardLibrary/pull/4250

  • ModelicaTest.MultiBody.Sensors.Distance2 There is no error reported.

  • ModelicaTest.MultiBody.Sensors.RelativeSensor There is no error reported.

tobolar avatar Mar 01 '24 14:03 tobolar

On 2024-06-18 I reviewed all issues with @tobolar, took decisions, and made actions, as reported in the top ticket description

casella avatar Jun 18 '24 15:06 casella

Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D

I tried to reduce the reference CSV to 0.4 s.

It nearly works in csv-compare. Unfortunately, it reports errors for each variable, because the tube extrapolates a bit in the future.

2024-07-02_differentStopTimes_CompareIssue

Proposal:

  • update reference file to stop at 0.4 s
  • accept regression failure, for now.
  • improve CSV-compare or agree on test-stop-time-annotation for future MSL development (CI)

GallLeo avatar Jul 02 '24 09:07 GallLeo

Agreed, as discussed during the 2024-07-02 MAP-Lib meeting

casella avatar Jul 03 '24 10:07 casella

For the above mentioned chaotic models, I cut the CSV files manually and added a Markdown note to the reference folders:

BevelGear1D https://github.com/modelica/MAP-LIB_ReferenceResults/blob/13debd9819062ea6ecc321538780f50717c11467/Modelica/Mechanics/MultiBody/Examples/Rotational3DEffects/BevelGear1D/readme.md

GearConstraint3 https://github.com/modelica/MAP-LIB_ReferenceResults/blob/13debd9819062ea6ecc321538780f50717c11467/ModelicaTest/MultiBody/Parts/Rotor1D/GearConstraint3/readme.md

GallLeo avatar Oct 21 '24 06:10 GallLeo

As far as I can tell, no items remain. If anyone disagrees, feel free to reopen.

Closing.

maltelenz avatar Feb 11 '25 12:02 maltelenz