hubot-markov icon indicating copy to clipboard operation
hubot-markov copied to clipboard

Add hubot brain storage module

Open jimcortez opened this issue 4 years ago • 2 comments

Add new hubot_brain storage option to use built-in hubot persistence.

This makes it so you can use other plugins to persist hubot (like hubot-redis-brain) as a whole.

You may want to consider making this default.

jimcortez avatar Jun 10 '20 20:06 jimcortez

This is setting the entire transition map into a single key-value pair within the brain's storage. The brain storage units that I've used (redis and postgresql) store that as a JSON blob in the associated backend. The markov model in the bot I run currently has about 2.5 million transition rows in its forward-chain model. Serializing and deserializing a JSON structure of that size on every received message would cause serious performance problems.

It's probably feasible to implement something like this backed by robot.brain.set and robot.brain.get with a key prefix instead. At least with the PostgreSQL brain variant I'm using, that would store each transition in a separate database row and do lookups with an indexed SELECT, which should be more scalable. (I'm not sure why I didn't do it that way to begin with - I think this module predates me finding/writing a PostgreSQL brain I was happy with?)

We should also alter the storage integration test suite to exercise this backend too.

smashwilson avatar Jun 11 '20 12:06 smashwilson

I actually use the redis integration of markov myself. I agree that in the long-term that having it in a separate store is probably wise. Just seemed like a good option to have for small bots, or for fun development.

You won't hurt my feelings if you close this PR.

jimcortez avatar Jun 12 '20 21:06 jimcortez