Lakhinder Walia

> > Use the new RNG for everything and update the tolerances for the verify tests that need it. > > We shouldn't increase the tolerance as we can miss...

The ranges of random number generation have been suitably expanded to provide maximum coverage. All the unit tests for Debug & Non-debug modes pass. A few tolerances had to be...

> The background I overheard for this PR is that the original random number generation did not adequately stress the kernels since it has a smaller subset of possible values....

> > The ranges of random number generation have been suitably expanded to provide maximum coverage. All the unit tests for Debug & Non-debug modes pass. A few tolerances had...

> > The original golden number (= 80) for tolerance was chosen in some heuristic way. > > The 80 was chosen because `80 * std::numeric_limits::epsilon()` is roughly 1e-6. This...

> This didnt find bugs in those cases. The only bug that was found was in cpu/layernorm.cpp, and you didnt fix the bug, you just change the test case so...

> I dont mean overflow in that sense, but since we are using all 24-bits of the mantissa so then when we do addition or multiplication we might need a...

> But this moves the tolerances in the wrong direction. We want to choose the test data so we can use much stricter tolerances. Strict tolerances coupled with poorly chosen...

> I'm OK with either 1 or 2. Option 3 would take refactoring a lot of tests so unlikely we'll be doing that. @CharlieL7, I have made all the changes...

(Not wanting to debate.. just adding some clarity for all.. just because of broad statements below) > > > somehow made look good by choosing compromised data. > > I...