rnn-from-scratch

rnn-from-scratch copied to clipboard

I am new to BPTT in Recurrent Neural Network. I made an alteration in the original article. Please enlighten and correct me.

ChongKim58

Caching the layer output and intermediates

1

Thanks for the wonderful write up! We can cache the layer output and intermediate to make this implementation fast.

hegman12

Hi, Why ds = dsv + diff_s what is the difference between diff_s and ds? I'am confused. thank you!

lonelydancer

你好！ 首先表示感谢，代码非常清晰易懂，唯一有点疑惑的是bptt的实现方式。 代码中是从前向后遍历T，用的是用t时刻的diff去算t-1 ~ 0 的梯度，是否使用t时刻diff由t+1时刻的梯度决定的方式会比较好一些，这样从后向前遍历T，一个循环就足够了。不过这样的话无法控制梯度传递的时间长度。

lanpay-lulu