server icon indicating copy to clipboard operation
server copied to clipboard

How to enable storage simplify

Open lucasjinreal opened this issue 1 year ago • 12 comments

I see the storage are implement using hooks, but I already have hooks how to make it more painless like out of box support for storage? the message a wipe out when server down, lost all messages inside. I even think this can be a builtin feature whcih can be enable by a simple flag.

lucasjinreal avatar Jan 03 '24 05:01 lucasjinreal

@lucasjinreal, you can add multiple hooks, they will all run. Since you're already using hooks, you can add additional ones like you did.

@dgduncan and @mochi-co are working on a new file-based configuration for the broker (#351), which will enable hooks via configuration file.

thedevop avatar Jan 03 '24 08:01 thedevop

I mean, the messages stores when program stop, and when start, it should resume old unsent messages, this should be a default bahavior

lucasjinreal avatar Jan 04 '24 02:01 lucasjinreal

That's what the storage hooks can provide, select the one that's appropriate for your backend: Badger, Bolt or Redis.

thedevop avatar Jan 04 '24 03:01 thedevop

@thedevop thanks, does the hooks provide in example with Badger can be used directly?

need install anything else and configuration anything else?

lucasjinreal avatar Jan 04 '24 03:01 lucasjinreal

I personally have not used persistent storage hooks. I would recommend taking a look at Badger example in README.

thedevop avatar Jan 04 '24 06:01 thedevop

@thedevop Can it work out of box without any further configurations? Enable the hook, and the unsent message can be automatically resumed?

lucasjinreal avatar Jan 04 '24 06:01 lucasjinreal

That's what PR #351 will enable.

thedevop avatar Jan 04 '24 18:01 thedevop

@thedevop Oh, no, this make things worse, it introduce annoying yaml files to run the lib, can't it set as default behavior to restore messages?

lucasjinreal avatar Jan 05 '24 02:01 lucasjinreal

@lucasjinreal the reason why this behavior isn't the default is that one of the most powerful parts of this project is that is an embeddable broker. The base assumption is that at default there is no guarantee that there is access to persistent storage between starts.

To enable something like badger it is as simple as err := server.AddHook(new(badger.Hook), &badger.Options{ Path: badgerPath, }) if err != nil { log.Fatal(err) }

where badgerPath is the relative location on your system for the badger files to be written to. Do you have a need that this does not fulfill?

As per your comment on #351 this is for when users want the ability to run the broker without worrying about actually embedding it themselves i.e run in a docker container by just passing a config.

dgduncan avatar Jan 07 '24 05:01 dgduncan

@dgduncan thanks, will this work?

err = server.AddHook(new(badger.Hook), &badger.Options{Path: "./uploads"})
	if err != nil {
		log.Fatal(err)
	}

uploads are relative to root of my exe project.

lucasjinreal avatar Jan 07 '24 05:01 lucasjinreal

@lucasjinreal That should work as long you have the correct permissions in order to read/write/delete from that location

dgduncan avatar Jan 08 '24 01:01 dgduncan

@lucasjinreal There's also an example for badger in the examples/persistence/badger folder. Let me know if you have any further questions! 🙂

mochi-co avatar Jan 11 '24 23:01 mochi-co

Closed due to inactivity

mochi-co avatar Mar 18 '24 03:03 mochi-co

Hi, I tested th exmaple, it;s not work in production,

It produced 4GB bin files but my server can not handle it, and I think the message amount didn't get that high.

Can u help users find a more useable way to dump the unsent messages?

lucasjinreal avatar Mar 18 '24 06:03 lucasjinreal

@lucasjinreal , you may need to use the latest version v2.6.0.

werbenhu avatar Mar 18 '24 10:03 werbenhu

@werbenhu Hi, if enable v2.6.0, can the issue resolved? Have u guys tested with it? And the cache file would be how large?

lucasjinreal avatar Mar 19 '24 02:03 lucasjinreal