RWKV-LM
RWKV-LM copied to clipboard
Probable mistake in Eq. 16 in the preprint
Thank you for the great work.
I have checked the code and the paper, and I believe that the "lerp" in Eq. 16 might actually be "ddlerp".
I also find the code uses ddlerp method. I also think Eq.19 also doesn't match the Table 7.
@zeyun-zhong You are correct, thanks for noticing this! We will update the paper.
@UEA-GCG Thank you for finding this error. I believe you are pointing out that the cumprod in Eq. 19 is indexed incorrectly. (It should go from j=i+1 to t-1, since it's really a reverse cumprod). Please let me know whether you see some other issue(s) as well with this formula.