recurrent-memory-transformer-pytorch icon indicating copy to clipboard operation
recurrent-memory-transformer-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

Question: How to set seq_len ?

Open pfeatherstone opened this issue 1 year ago • 1 comments

What is a good number for seq_len ? What are the trade-offs for shorter or longer seq_len? Like, why can't seq_len==1 ? Infinite recurrence is infinite recurrence no matter what the value is right?

pfeatherstone avatar Aug 08 '23 07:08 pfeatherstone

@pfeatherstone you still want seq_len to be as long as possible. i would keep it at 2048 - 4096

all the issues that plague RNNs will come back (vanishing gradients, retaining past information across steps, etc)

lucidrains avatar Aug 09 '23 14:08 lucidrains