llm.c
llm.c copied to clipboard
Fixed a TODO to calculate the max value neatly and use inv sum trick
There was a TODO in the softmax_forward function to calculate the maxval neatly (float maxval = -10000.0f; // TODO something better
). You can take the first value as the max and then iterate over the rest of the values in the loop to calculate the max using the fmaxf
function.
In the normalization loop, we can multiply rather than divide as this can be faster on some hardware.
It makes me a bit uncomfortable that this fails when V = 0. But when V = 0 something went really wrong. I'll probably merge a variation of this ty
@sirvan3tr but you're dividing AND multiplying, instead of just dividing, or am I missing something?