char-rnn
char-rnn copied to clipboard
karpathy
Scoring Functionality
It would be nice to add in a third script score.lua which would take a trained model and an example (or many examples) and provide log probabilities of the character sequence according to the model.
I've started looking into adding this based on the sampling code, but have to learn Lua and Torch in the process. I imagine it would be a pretty simple addition for someone who knows what they're doing?
I'd look at eval_split function in the main training code instead of the sampling script.
A few follow up questions:
- I see that we obtain a per-character log-likelihood as the output of the rnn:
prediction = lst[#lst]. For scoring purposes, If the input sequence is 'hello', we would feed in 'h', get the LP for 'e' and add that to some accumulator. From there, forward 'e', get the LP for 'l', accumulate that, and so on. Is this correct? - If the above is correct, how does one determine the initial distribution of probabilities for a given character at the start of a sequence.
- And finally, how does one reset the state of the RNN for scoring multiple sequences?
Thanks for your time!
@ryaneleary Did you ever get this scoring script working? I would be very interested to see/use it. @karpathy I saw your talk in Whitechapel the other Monday. Thanks for a great introduction to RNNs and LSTMs.
@JohnReid I did get it working. If you look at the closed PR (#61) you can see the implementation I used. It hasn't been rebased on any of the changes that have been going on in the main char-rnn code, so it may require some cleanup/modifications if you need it to run against models built by the latest version. Otherwise, it seemed to work fine for my task (scoring a given string against two different models and then compute a LLR).
@ryaneleary I saw this and I have a slightly modified version working now. Many thanks.
@JohnReid I've added you as a collaborator on my fork of the repo. If you made improvements to the script and wouldn't mind pushing them to a branch, I'd greatly appreciate it. My attention has been elsewhere recently, but I have definite plans to be using this in the near future.