RWKV-LM icon indicating copy to clipboard operation
RWKV-LM copied to clipboard

Probable mistake in Eq. 16 in the preprint

Open zeyun-zhong opened this issue 1 year ago • 2 comments

Thank you for the great work.

I have checked the code and the paper, and I believe that the "lerp" in Eq. 16 might actually be "ddlerp".

zeyun-zhong avatar Apr 21 '24 20:04 zeyun-zhong

I also find the code uses ddlerp method. I also think Eq.19 also doesn't match the Table 7.

UEA-GCG avatar Apr 23 '24 11:04 UEA-GCG

@zeyun-zhong You are correct, thanks for noticing this! We will update the paper.

@UEA-GCG Thank you for finding this error. I believe you are pointing out that the cumprod in Eq. 19 is indexed incorrectly. (It should go from j=i+1 to t-1, since it's really a reverse cumprod). Please let me know whether you see some other issue(s) as well with this formula.

SmerkyG avatar Apr 30 '24 21:04 SmerkyG