Infermo icon indicating copy to clipboard operation
Infermo copied to clipboard

implemented std_grad in backward.mojo

Open ManishAradwad opened this issue 2 years ago • 3 comments
trafficstars

@TilliFe @andresnowak please verify the implementation #9

ManishAradwad avatar Nov 06 '23 16:11 ManishAradwad

From what i understand, the only thing that is incorrect is this a.grad.simd_load[_nelts](idx_a) * std_derivativeinstead of a multiplication it should be a sum, we have to accumulate the gradient in the a Tensor and doing the chain rule in this operation std_derivative * b.grad.load(idx_b)

andresnowak avatar Nov 06 '23 21:11 andresnowak

Thanks @ManishAradwad :). I will merge this as soon as I am back. Leave this PR open.

TilliFe avatar Nov 07 '23 07:11 TilliFe

From what i understand, the only thing that is incorrect is this a.grad.simd_load[_nelts](idx_a) * std_derivativeinstead of a multiplication it should be a sum, we have to accumulate the gradient in the a Tensor and doing the chain rule in this operation std_derivative * b.grad.load(idx_b)

Right! I'll correct it

ManishAradwad avatar Nov 07 '23 07:11 ManishAradwad