Help with LSTM basics in synaptic.js

[r3ndd]

I looked through the Wikipedia Reading example for help, but I couldn't actually get the example to run and the code itself is hard to follow as someone who hasn't used LSTM networks before. I would like to make a simple LSTM somewhat like the Wiki Reader, but I'm not sure how to both train the network so that it incorporates past input and then activate it in a way that it utilizes previous activations.

I'd like to see a basic example of the training and activation of the LSTM architect. Thanks.

[andymakhk]

Hello, I was also looking into the same area.

There are actually 2 LSTM examples in this project:

1. Discrete Sequence Recall
2. Read From Wikipedia

The first one is easier to understand, but it is still a guided / supervised learning. The wikipeida one is sort of like an unguided / unsupervised learning, which I think more people is interest in (maybe!?).

One more thing about the wikipeida example is, the web demo actually doesn't work at the moment (maybe web traffic load is too heavy?). It requires a PHP Proxy page to crawl articles from wikipeida; convert them into JSON; and feed them into the browser to work with synaptic.js

In order to experiment on that, you may want to install your own copy of PHP server and host that proxy page locally on the PC: source code in here: https://github.com/cazala/synaptic-wikipedia

[r3ndd]

@andymakhk I might be wrong about this, but it is my understanding that RNNs such as LSTM networks use some form of memory in time steps, where previous inputs influence future outputs. How do I utilize this in Synaptic, both in training and in testing? And furthermore for testing, how do I reset the memory of the network so that I can test different data that is its own independent series?

[mjp0]

I've to agree that synaptic would really benefit from a bit more documentation.

For example in DSR code there's part:

var results = LSTM.trainer.DSR({
    targets: targets,
    distractors: distractors,
    prompts: prompts,
    length: length,
    iterations: 250000,
    rate: .17
});

This is very confusing because all the other docs talk about how you only need trainer.train() and training set in certain format to use all architectures. I'm currently playing with LTSM but can't get training to start - it just hangs and does absolutely nothing.

[vkaracic]

This might help: https://github.com/vkaracic/JARVIS/blob/master/static/js/LSTM.js It's part of a simple UI that I'm working on. It's the code for creating and training an LSTM network.

[r3ndd]

@vkaracic Thanks! This was actually very helpful. The only thing I'm stuck on now is how do I reset the network's memory?

For instance, if I'm generating a sentence letter-by-letter based on the letters it has already generated, how would I go about resetting it so that I can generate a new sentence without the letters from the previous sentence affecting it?

[basickarl]

The documentation is poor indeed.

[basickarl]

How would one train the LSTM to do a time series prediction on the following array:

array = [
    0,
    0,
    0,
    1,
    0,
    0,
    0,
    1,
];

To be honest a simple example like this should exist.