Sarkars/batchnorm update

[sayantan-nervana]

Opening a PR for future ngraph BN API change: https://github.com/NervanaSystems/ngraph/pull/2046/files

[sayantan-nervana]

TESTNOW

[sayantan-nervana]

TESTNOW

[sayantan-nervana]

Fail message:

self = <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>

    def test_tanhgrad_2d(self):
        y = constant_op.constant(
            self.generate_random_numbers(30, 1.0, 10.0), shape=[10, 3])
        y_delta = constant_op.constant(
            self.generate_random_numbers(30, 0.0, 10.0), shape=[10, 3])
    
        out = tanh_grad(y, y_delta)
    
        def run_test(sess):
            return sess.run(out)
    
>       assert np.allclose(
            self.with_ngraph(run_test), self.without_ngraph(run_test))
E       assert False
E        +  where False = <function allclose at 0x7f736018d488>(array([[-5.6790155e+02, -1.0593641e+02, -5.1666357e+02],\n       [-9.5448242e+0...e+02],\n       [-4.1972357e+02, -3.4175458e+02, -1.0048141e+02]], dtype=float32), array([[-5.6790155e+02, -1.0593641e+02, -5.1666357e+02],\n       [-9.5448242e+0...e+02],\n       [-4.1972357e+02, -3.4175458e+02, -1.0048141e+02]], dtype=float32))
E        +    where <function allclose at 0x7f736018d488> = np.allclose
E        +    and   array([[-5.6790155e+02, -1.0593641e+02, -5.1666357e+02],\n       [-9.5448242e+0...e+02],\n       [-4.1972357e+02, -3.4175458e+02, -1.0048141e+02]], dtype=float32) = <bound method TestTanhGradOp.with_ngraph of <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>>(<function run_test at 0x7f72300a0a28>)
E        +      where <bound method TestTanhGradOp.with_ngraph of <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>> = <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>.with_ngraph
E        +    and   array([[-5.6790155e+02, -1.0593641e+02, -5.1666357e+02],\n       [-9.5448242e+0...e+02],\n       [-4.1972357e+02, -3.4175458e+02, -1.0048141e+02]], dtype=float32) = <bound method TestTanhGradOp.without_ngraph of <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>>(<function run_test at 0x7f72300a0a28>)
E        +      where <bound method TestTanhGradOp.without_ngraph of <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>> = <test_tanhgrad.TestTanhGradOp object at 0x7f72180e2b50>.without_ngraph

test_tanhgrad.py:45: AssertionError
=============== 1 failed, 79 passed, 51 skipped in 4.62 seconds ================

[jianyinglang]

One more comment: in ngraph core, it requires the input dimension >=2. I think it might be good if we add this as a confirmation constraint.