Fixing last failed test for Release build

[khuck]

The release build has one last failing test: http://omega.nic.uoregon.edu:8020/#/builders/2/builds/167/steps/16/logs/stdio

I tracked this down to temporaries output.tensor() and target.tensor() going out of scope in cat_cross_operation::cat_cross3d(): https://github.com/STEllAR-GROUP/phylanx/blob/64540c34dcec64530cc5c9239fb36144b1302bee/src/plugins/keras_support/categorical_crossentropy_operation.cpp#L246-L297

I'll submit a pull request for that issue (see #1188).

However, after fixing that crash, the python code fails assertions:

AssertionError: [[ 0.96681002  2.18922338  3.41163674  4.6340501 ]
 [ 5.85646346  7.07887682  8.30129018  9.52370354]
 [10.7461169  11.96853026 13.19094362 14.41335698]]
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /var/lib/buildbot/slaves/phylanx/x86_64-gcc7-release/build/tests/unit/python/execution_tree/categorical_crossentropy.py(79)<module>()
-> assert np.all(np.abs(v1 - v2) < 1.0e-13), v1 - v2
(Pdb) p v1
array([[ 3.,  7., 11., 15.],
       [19., 23., 27., 31.],
       [35., 39., 43., 47.]])
(Pdb) p v2
array([[ 2.03318998,  4.81077662,  7.58836326, 10.3659499 ],
       [13.14353654, 15.92112318, 18.69870982, 21.47629646],
       [24.2538831 , 27.03146974, 29.80905638, 32.58664302]])
(Pdb) list
 74  	    o = 3 * np.ones(10)
 75  	    o[4] = 2
 76  	
 77  	    v1 = cc(t, o, flag, -1)
 78  	    v2 = cat_cross(t, o, flag)
 79  ->	    assert np.all(np.abs(v1 - v2) < 1.0e-13), v1 - v2
 80  	
 81  	    t = np.linspace(1.0e-9, 10e-9, 10)
 82  	    t[2] = 10
 83  	    o = 3 * np.ones(10)
 84  	    o[4] = 2
(Pdb) 

[hkaiser]

Which test is failing, concretely? Could you elaborate, please?

[khuck]

I am trying to hunt that down, which isn't easy because this program requires a mix of gdb and pdb since it's a python script test case.

[khuck]

Which test is failing, concretely? Could you elaborate, please?

it is the tests.unit.python.execution_tree.categorical_crossentropy test, if that's what you mean

[khuck]

@hkaiser Specifically, the temporaries that appear to go out of scope are those bound by the lambda and sum3d() calls here: https://github.com/STEllAR-GROUP/phylanx/blob/64540c34dcec64530cc5c9239fb36144b1302bee/src/plugins/keras_support/categorical_crossentropy_operation.cpp#L284-L290

So while I've fixed the crash, the assertion fails on line 125 of tests/unit/python/execution_tree/categorical_crossentropy.py in the 3d test when axis is -1 and flag is True. https://github.com/STEllAR-GROUP/phylanx/blob/64540c34dcec64530cc5c9239fb36144b1302bee/tests/unit/python/execution_tree/categorical_crossentropy.py#L111-L125

I get these matrices:

(Pdb) p v1
array([[ 3.,  7., 11., 15.],
       [19., 23., 27., 31.],
       [35., 39., 43., 47.]])
(Pdb) p v2
array([[ 2.03318998,  4.81077662,  7.58836326, 10.3659499 ],
       [13.14353654, 15.92112318, 18.69870982, 21.47629646],
       [24.2538831 , 27.03146974, 29.80905638, 32.58664302]])
(Pdb) p flag
True
(Pdb) p axis
-1
(Pdb) p o
array([[[0.4, 0.5],
        [0.6, 0.7],
        [0.8, 0.9],
        [1. , 1.1]],

       [[1.2, 1.3],
        [1.4, 1.5],
        [1.6, 1.7],
        [1.8, 1.9]],

       [[2. , 2.1],
        [2.2, 2.3],
        [2.4, 2.5],
        [2.6, 2.7]]])
(Pdb) p t
array([[[ 1.,  2.],
        [ 3.,  4.],
        [ 5.,  6.],
        [ 7.,  8.]],

       [[ 9., 10.],
        [11., 12.],
        [13., 14.],
        [15., 16.]],

       [[17., 18.],
        [19., 20.],
        [21., 22.],
        [23., 24.]]])