llm.c icon indicating copy to clipboard operation
llm.c copied to clipboard

Fixed a TODO to calculate the max value neatly and use inv sum trick

Open sirvan3tr opened this issue 10 months ago • 2 comments

There was a TODO in the softmax_forward function to calculate the maxval neatly (float maxval = -10000.0f; // TODO something better). You can take the first value as the max and then iterate over the rest of the values in the loop to calculate the max using the fmaxf function.

In the normalization loop, we can multiply rather than divide as this can be faster on some hardware.

sirvan3tr avatar Apr 11 '24 11:04 sirvan3tr

It makes me a bit uncomfortable that this fails when V = 0. But when V = 0 something went really wrong. I'll probably merge a variation of this ty

karpathy avatar Apr 11 '24 15:04 karpathy

@sirvan3tr but you're dividing AND multiplying, instead of just dividing, or am I missing something?

dagelf avatar Apr 18 '24 19:04 dagelf