PolyMath icon indicating copy to clipboard operation
PolyMath copied to clipboard

Some tests are failing randomly from time to time

Open olekscode opened this issue 3 years ago • 8 comments

This is probably because they don't set the random state. For example:

  • PMQRTest >> testMPInverse

olekscode avatar May 05 '22 13:05 olekscode

I've begun looking into this. Could you clarify what should happen when random state is set, please? As an alternative, I wondered whether a simpler test could be written. The current one is difficult for me to follow.

hemalvarambhia avatar Jun 20 '22 20:06 hemalvarambhia

I've begun carefully refactoring the testMoorePenroseInverse test to enhance clarity (Four Elements of Simple Design).

To effectively test the Moore Penrose Inverse computation, is it necessary to set random state, @olekscode ? Due to my complete ignorance I am unsure of why.

Thank you.

hemalvarambhia avatar Jun 26 '22 22:06 hemalvarambhia

Another test that's failing is

PMNumericalMethodsTestCase
[1496](https://github.com/PolyMathOrg/PolyMath/runs/7117052304?check_suite_focus=true#step:4:1497)
 ✗ #testOptimizeOneDimension (12ms)

hemalvarambhia avatar Jun 29 '22 20:06 hemalvarambhia

Another test that's failing erratically:

PMGeneralFunctionFitTest
[1484](https://github.com/PolyMathOrg/PolyMath/runs/7119999338?check_suite_focus=true#step:4:1485)
 ✗ #testMaximumIterationsProbabilistic (1009ms)

hemalvarambhia avatar Jun 29 '22 20:06 hemalvarambhia

testQRFactorisation in PMQRTest fails erratically

hemalvarambhia avatar Jun 29 '22 20:06 hemalvarambhia

@olekscode I think I managed to corner the input matrix that is responsible for the erratically failing test:

testInputOfErraticFailure

| a |
a := PMSymmetricMatrix rows:
    #( #( 0.41929313699681925 0.05975350554089691
          0.2771676258543356 0.35628773381760703 )
       #( 0.05975350554089691 0.12794227252152854
          0.3257742693302102 0.28814463284245906 )
       #( 0.2771676258543356 0.3257742693302102 0.8468441832097453
          0.9101872061892353 )
       #( 0.35628773381760703 0.28814463284245906
          0.9101872061892353 0.5163744224777326 ) ).

self assert: (a mpInverse closeTo: a inverse)

This fails for the same reason as highlighted during the CI run. Thus we have a deterministic test we can use to corner the failure much more.

If you'd like to investigate with me, you can find this in investigate-erratic-test

Does the above matrix relate in some way to your hypothesis of not setting the random state.

My investigate points to an array (3 4 3 nil) - the nil appears to be the cause.

hemalvarambhia avatar Jul 03 '22 20:07 hemalvarambhia

Another test that's failing is

PMNumericalMethodsTestCase
[1496](https://github.com/PolyMathOrg/PolyMath/runs/7117052304?check_suite_focus=true#step:4:1497)
 ✗ #testOptimizeOneDimension (12ms)

I confirm this random failure:

PMNumericalMethodsTestCase ✗ #testOptimizeOneDimension (12ms) TestFailure: Assertion failed PMNumericalMethodsTestCase(TestAsserter)>>assert:description:resumable: PMNumericalMethodsTestCase(TestAsserter)>>assert:description: PMNumericalMethodsTestCase(Object)>>assert: PMNumericalMethodsTestCase>>testOptimizeOneDimension ...assert: (maximum - 5) abs < 1.0e-6 PMNumericalMethodsTestCase(TestCase)>>performTest

jecisc avatar Nov 29 '22 15:11 jecisc

This is probably because they don't set the random state. For example:

* PMQRTest >> testMPInverse

I fixed this one and all related to random matrix and vectors: https://github.com/PolyMathOrg/vector-matrix/commit/5fb38e94b5b2a74eed33a173bbc7a0f4931df66e

jecisc avatar Mar 13 '23 15:03 jecisc