CUDA-Learn-Notes icon indicating copy to clipboard operation
CUDA-Learn-Notes copied to clipboard

verified publisher icon DefTruth
→

Metadata

🎉 Modern CUDA Learn Notes with PyTorch: fp32, fp16, bf16, fp8/int8, flash_attn, sgemm, sgemv, warp/block reduce, dot, elementwise, softmax, layernorm, rmsnorm.

Results 6 CUDA-Learn-Notes issues
Sort by recently updated
recently updated
newest added
trafficstars

resources

1 comment

![cuda-learn-note](https://github.com/DefTruth/CUDA-Learn-Note/assets/31974251/882271fe-ab60-4b0e-9440-2e0fa3c0fb6f)

DefTruth avatar DefTruth

stale

__threadfence() 作用

3 comment

佬有测试过 0x09 softmax 中的 `__threadfence()`吗?这个好像没办法达到grid级别线程之间的同步.

zbt78 avatar zbt78

您好，请教一个关于代码中reduce相关的问题

3 comment

1. `sum = warp_reduce_sum(sum);` 2. `if(warp==0) sum = warp_reduce_sum(sum);` 0x03 warp/block reduce sum/max 、0x09 softmax, softmax + vec4 做final sum的时候，用的是第一种形式 0x04 block all reduce + vec4 而用的是第二种形式 我的理解是，最后final sum的时候是不是应该用第二种形式？最后都集中在第一个warp束中。 感谢！

Ss-shuang123 avatar Ss-shuang123

[GELU] Add f32/x4, f16/x2/x8/x8pack kernel.

1 comment

Saw the mention of GELU in the issue, so I worked on it. There is no GELU implementation in torch for half precision (the reason is explained in readme.md), so...

bear-zd avatar bear-zd

Kernel Trace issue

## TODO - [ ] swish kernel - [ ] gelu kernel - [ ] RoPE kernel - [x] pack elementwise_add - [x] pack sigmoid - [x] pack relu -...

DefTruth avatar DefTruth

layer norm实现

2 comment

readme里面layer norm的实现是不是batch norm的啊

zbt78 avatar zbt78

← Metadata

1.2k
Stars
133
Forks
Watchers

Owner

verified publisher icon DefTruth

Metadata

🎉 Modern CUDA Learn Notes with PyTorch: fp32, fp16, bf16, fp8/int8, flash_attn, sgemm, sgemv, warp/block reduce, dot, elementwise, softmax, layernorm, rmsnorm.

Back