neataptic
neataptic copied to clipboard
[question] How `clear` works?
Hi, can you please explain how the clear
works? The docs says
clear - If set to true, will clear the network after every activation. This is useful for training LSTM's, more importantly for timeseries prediction. Default: false
I'ts not very clear what's the every activation
means. Does it mean that it will be cleared after every example? If so - how would it be trained to predict - let's say string parsing.
Let's say we want to parse strings with LSTM:
'two' => 2
'one' => 1
'seven' => 7
Every single example would be converted into group - a sequence of multple onehots.
In my understanding clear
should be called not for every activation (every onehot) but for every group (mutliple onehots). See example below:
(0 and 1 are random, just to demonstrate)
// first example 'two' => 2
clear
[0, 0, 1, 0] => [0, 0] // t => nothing
[0, 1, 0, 0] => [0, 0] // w => nothing
[1, 0, 0, 0] => [0, 1] // o => 2
// second example 'one' => 1
clear
[0, 0, 1, 0] => [0, 0] // o => nothing
[0, 1, 0, 0] => [0, 0] // n => nothing
[1, 0, 0, 0] => [0, 1] // e => 1
But it seems like what the docs are saying is the following
// first example 'two' => 2
clear
[0, 0, 1, 0] => [0, 0] // t => nothing
clear
[0, 1, 0, 0] => [0, 0] // w => nothing
clear
[1, 0, 0, 0] => [0, 1] // o => 2
// second example 'one' => 1
clear
[0, 0, 1, 0] => [0, 0] // o => nothing
clear
[0, 1, 0, 0] => [0, 0] // n => nothing
clear
[1, 0, 0, 0] => [0, 1] // e => 1
So, how does the clear
works? Is it per onehot or per group of onehots?
Actually I figured it out - you need to set clear: false
and add one more onehot for termination signal (the neural network should be smart enough to figure out to clear itself when see it), like that:
// first example 'two' => 2
[0, 0, 1, 0, 0] => [0, 0] // t => nothing
[0, 1, 0, 0, 0] => [0, 0] // w => nothing
[1, 0, 0, 0, 0] => [0, 1] // o => 2
[0, 0, 0, 0, 1] => [0, 0] // special onehot signalling to do clear
// second example 'one' => 1
[0, 0, 1, 0, 0] => [0, 0] // o => nothing
[0, 1, 0, 0, 0] => [0, 0] // n => nothing
[1, 0, 0, 0, 0] => [0, 1] // e => 1
[0, 0, 0, 0, 1] => [0, 0] // special onehot signalling to do clear