Precompute the scaling factor in gelu_forward and gelu_backward

[ryanmcdermott]

Thank you so much for the amazing repo!

The scaling factor in gelu_forward and gelu_backward is computed each time the functions are run: float s = sqrtf(2.0f / M_PI);

This can be precomputed ahead of time and stored as a constant.

[ayushanshul07]

Is this good enough?

[andylolu2]

Doesn't it get optimised away by the compiler anyway? (Haven't actually checked though) Plus pointwise operations are bandwidth-limited anyway, so adding/removing a few flops shouldn't make a difference.

[karpathy]

fixed

[ayushanshul07]

@ryanmcdermott Isn't #defining constant is just replacement and the actual calculation is still repeated? https://github.com/karpathy/llm.c/commit/f26cf00a615e4f643a3660071e4ef86010045db7