PENG Bo

I can do first and second manually. So now it's about DeepSpeed support :)

With GUI [https://withablink.coding.me/goPolicyNet/](https://withablink.coding.me/goPolicyNet/) : 

我也想提交，你改一改吧，不要完全copy了，哈哈。

你好，可以用 wandb

Thanks. Will do.

> Hello, I am trying out RWKV with audio modality and when I set T_MAX>>1000, it throws this error: Reduce B_GROUP_FORWARD and B_GROUP_BACKWARD.

> Also, it seems FP16 doesn't work out-of-the-box. Could you suggest changes to make it work? You can move the FFN to FP16 first :)

> I did that, but now it gives `Illegal memory accessed` at `k.continguous()` in the forward of the TimeMix. Works fine in fp32. The CUDA code assumes a tensor element...

> @BlinkDL Can you please point out where we need to make a change to the code to reduce the tensor element from 4 bytes to 2 bytes? Thanks a...

你好，可以试试传统方法，但还有一个办法，RWKV 的 hidden state 很小（请看 https://github.com/BlinkDL/RWKV-v2-RNN-Pile/blob/main/src/model.py 的 .xx .aa .bb ），可以试试直接加个线性层输出。试试用 .xx 和 .aa / .bb 作为线性层的输入。