Shao Tang comments

Repositories
Issues
Comments

Results 14 comments of


                                            Shao Tang

Include the online softmax CPU code and a fully parallelized GPU kernal

Updated with a full parallel kernel.

Reduce the duplicate computation in crossentropy and include the V padding for crossentropy kernel

> Isn't the `if` necessary for safety? Correct, too focused on the thread divergence and forgot about the basics : ) Updated the PR

Reduce the duplicate computation in crossentropy and include the V padding for crossentropy kernel

Include a kernel taking padded probs.

Error: make: *** [Makefile:203: train_gpt2cu] Error 255

1. upgrade nvcc to 12.4. 2. check the computation capability of the GPU card, in the source code include/cuda_bf16.h (or hpp). You might see ``` #if defined(__CUDACC__) && (!defined(__CUDA_ARCH__) ||...