responsible-ai-toolbox icon indicating copy to clipboard operation
responsible-ai-toolbox copied to clipboard

Use fused multiply-add to apply `grad_scale` and `grad_bias`

Open rsokl opened this issue 2 years ago • 0 comments

https://pytorch.org/docs/stable/generated/torch.add.html

>>> a = torch.randn(4)
>>> a
tensor([ 0.0202,  1.0985,  1.3506, -0.6056])

>>> b = torch.randn(4)
>>> b
tensor([-0.9732, -0.3497,  0.6245,  0.4022])
>>> c = torch.randn(4, 1)
>>> c
tensor([[ 0.3743],
        [-1.7724],
        [-0.5811],
        [-0.8017]])
>>> torch.add(b, c, alpha=10)
tensor([[  2.7695,   3.3930,   4.3672,   4.1450],
        [-18.6971, -18.0736, -17.0994, -17.3216],
        [ -6.7845,  -6.1610,  -5.1868,  -5.4090],
        [ -8.9902,  -8.3667,  -7.3925,  -7.6147]])

rsokl avatar Apr 26 '22 16:04 rsokl