Shao Tang issues

Results 33 issues of


                                            Shao Tang

Reduce the duplicate computation in crossentropy and include the V padding for crossentropy kernel

- introduce `const int bt = b * T + t` to reduce the duplicated computation. [Running 5+ trials on 3070, no obvious improvement in the performance.] - Include the...

small refactor for better readability

Include a factor in add bias kernel of matmul for perf tuning

A larger `thread_reuse_factor` reduces the number of threads launched while increasing the per-thread load. Depending on the value of `B * T * OC` and the GPU card, it is...

Fix the bug that yields cpu, gpu results mismatch in crossentropy_softmax_backward.cu

With master branch ``` ❯ nvcc -O3 --use_fast_math crossentropy_softmax_backward.cu -o crossentropy_softmax_backward ❯ ./crossentropy_softmax_backward 1 Using kernel 1 Checking block size 32. -0.438539 -0.438539 0.136152 0.136152 -0.364946 -0.364946 -0.384722 -0.384722 -0.530659...

const-fix crossentropy_softmax_backward.cu

https://github.com/karpathy/llm.c/issues/147

const-fix classifier_fused.cu

https://github.com/karpathy/llm.c/issues/147 script execution normally after fix

Fix the 0.0f initialization for double

Makefile type fix: move gelu_backward to backward block

Rewrite the encoder_forward float4 kernel with pack128

No notice of change in performance after the changing from float4 to pack128.

float4 with better vectorization for adamw.cu

On 3070, Kernel 2 time gpu 0.0799 ms time cpu 0.0168 ms Kernel 3 time gpu 0.0780 ms time cpu 0.0166 ms