Add ability to store message IDs persistent

[patcon]

So that between Slack instance, we can preserve threading (https://github.com/42wim/matterbridge/issues/529) and send emojis (https://github.com/42wim/matterbridge/issues/526) with more consistent behaviour :)

Thoughts? I could be pretty simple, and totally optional for people who choose to stay utterly stateless. With matterbridge-heroku, adding persistence of relationships could be as simple as:

$ heroku addons:create heroku-redis:hobby-dev

Curious your thoughts on this very Slack-specific addition

[42wim]

Quick thoughts, redis seems like overkill. (and isn't really made for persistency afaik) I'm more for using something like boltdb (https://github.com/etcd-io/bbolt) (simple embedded database).

As long as it's optional (and simple) it's ok for me.

[patcon]

Bolt seems cool but I'd really prefer something that continues to work easily on heroku, where I have a few instances running :)

I'll think on it. I think you're right about redis not being for persistence. Perhaps postgres. Free plan has 10,000 rows, which exactly matches how much history our slack instance has anyhow...!

[42wim]

Why do you think bolt wouldn't work?

[patcon]

Unless I'm misunderstanding, no heroku service addons offer bolt. Heroku is stateless without a file system, so it needs an add-on provider for its ability to have services (databases, caches, etc) as push-button

[patcon]

Ok, so on further investigation, redis actually seems a step in the right direction. I don't think we need full persistency perhaps, but we need to be able to query based on more than primary key (original message timestamp), which is only what the lru cache allows.

So assume an example of wanting to rethread messages. We instead need something that would allow us to query for a mID value in BrMsgID from the lru cache (using a ThreadTimestamp from thread reply ), and get the key (which is the original Timestamp). We could then know all the other messages to update from there.

For now, I suppose I could just skirt the change by looping through the lru cache. Curious your thoughts on a larger solution though

[42wim]

We should use https://github.com/abronan/valkeyrie if this gets implemented.

[patcon]

perrrrfect. I'll prob work on this next. Getting to crunch time on another project, so might not be as soon as I'd like

[patcon]

Potentially helpful link: https://medium.com/@nevio/proper-way-how-to-marshal-and-unmarshal-golang-structs-within-the-redis-cache-855423902787

Was curious about using msgpack (which redis-cache golang lib uses), but seems marshalling as json should work fine: https://www.reddit.com/r/golang/comments/476fdi/what_is_best_serialization_format_for_boltdb_for/

[42wim]

The valkeyrie library should handle all that. There shouldn't be any redis or bolt specific code in matterbridge.

[patcon]

No, valkeyrie doesn't do marshalling -- just accepts byte slices as far as I can tell. Anyhow, I have a code spike here: https://github.com/42wim/matterbridge/compare/master...patcon:valkeyrie-persistence?expand=1

Busy week coming up, so that's likely all I'll get done, but anyone is welcome to squash my commits and then fix it up :)

EDIT: Works for edits and threads, though file cache is still using a more rudimentary connection approach from earlier in my experimentation

[42wim]

@patcon okay nice! Love your enthusiasm, but before adding any new PR's could you finish the ones that are already opened? :)

[nylen]

+1 for this feature, and also for the ability to use a local/embeddable database that is created automatically when needed. Not requiring any other services to be running on the machine is a big advantage of this program.

[sas1024]

May be try to use gokv - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more) ?

[sas1024]

@patcon hello! Can you share your code with valkeyrie implementation for persistent message store? I want to implement this thing and any help will be useful :)

[sas1024]

@42wim hello! Can you share your thoughts on implementing this feature? I want to realize this feature.

For now in code i see:

• HashiCorp LRU cache usage.
• This cache initialized and used in multiple places (in gateway, slack, discord, mattermost, rocketchat packages)

IMHO there is two possible implementations for persistent storing message IDs:

1. Completely replace LRU cache with valkeyrie.
2. Use LRU cache and valkeyrie together. If no caching backend was configured, then will be used LRU cache. In this implementation we need to create abstraction between LRU cache and valkeyrie.

And we can initialize cache once and use it in all required places.

[sas1024]

@42wim ping? :)

[patcon]

A-ha! I realized that I transferred the repo to another org, and my personal fork was a more recent one without my old branches. The code is really old, but it's here 🎉 ~~https://github.com/42wim/matterbridge/compare/master...patcon:valkeyrie-persistence?expand=1~~ https://github.com/g0v-network/matterbridge/tree/valkeyrie-persistence (merge conflict now, so not a clean diff, but still can check it out)

Anyhow, good luck and godspeed @sas1024 ! I'm rooting for you!

[celine-m-s]

Hey there! Any updates on this? I just implemented Matterbridge to synchronize messages between Slack and Mattermost but it tends to crash twice a week. This is not a problem in itself, except it breaks all threads (which is quite annoying) so I'd be very happy if persistent threads were in the roadmap. By the way, thanks for all the hard work! :clap:

[yousefmansy1]

@sas1024 did you ever get anywhere on writing a solution for this feature? I'd be willing to trying my hand at it as well.

The code link @patcon posted does not appear to exist anymore either, hoping you have it saved somewhere?

https://github.com/42wim/matterbridge/compare/master...patcon:valkeyrie-persistence?expand=1

[sas1024]

@yousefmansy1 I'm sorry, but I haven't started working on this feature yet :( It well be fantastic, if you will try to realize this feature.

[yousefmansy1]

Ok I see, no worries. Do you have the code @patcon posted in https://github.com/42wim/matterbridge/issues/541#issuecomment-1107951615?

It seems to be a dead link for me.

[sas1024]

No, I don't have this code, this link was broken several months. May be @patcon can share this code again?

[yousefmansy1]

No worries, thanks for the response. I'll probably try to take a look using the gokv thing you mentioned.

[patcon]

oy... i'm sorry, i'm seeing that I must have confused to clipboard, and shared the same link again. It can't be merged anymore without conflict resolution, but here's the branch with the work: https://github.com/g0v-network/matterbridge/tree/valkeyrie-persistence