[Feature] Outbox Archival / Management

[preardon]

The outbox can tend to get quite big, some people decide to implement their own solutions, it would be a lot better if Brighter could manage these

• Ability to Archive to another Table / Store ( Configurable)
  • Initially Same outbox provider
  • Longer term ability to archive to a different store
• Ability to truncate old messages (Configurable)

[iancooper]

Agreed, we originally assumed that folks would handle this themselves i.e. write their own routines to run a scheduled job to do cleanup. We might want to offer to install this for you. I guess some design decisions are:

• Treat this as DDL that we use to configure a scheduled process that does cleanup in your Db
• Have Brighter help do this clean up? Does this run as a separate process to your producer? As a background process in any producer?
• Do we offer a choice between archive and delete for ever? Some folks may not choose to archive but delete.

So, agree it's "out-of-the-box" but we need to figure out the requirements I think.

[preardon]

in my head I would do this similiarly to sweeping,

• Have the command Processor configurable (archive / delete) and a method that takes care of it (I'm thinking similar to Implicit Clear)
• Add a Hosted service the the hosting package so that this can be run in its own process

This keep everything in the Brighter Domain

[iancooper]

@preardon

Have the command Processor configurable (archive / delete) and a method that takes care of it (I'm thinking similar to Implicit Clear)

Normally you would create a policy that lets you say: archive at X days, delete from archive at X days; archive at X days; don't delete; don't archive delete at X days, no archive to delete etc. So you would want to define a lite DSL - think Polly - that lets you define that.

If we define as a separate host, we might want to think about whether we want to start a general 'management' host. As we begin to think about some of these issues it could be useful to have a management host, with an API, command line that talks to the API etc. For example, you might kick this off with a policy that can read its variables from config, environment etc, but also allow someone to "do it now" via a call to an API that a command line gives you.

I don't think we need a distributed lock here, but that may be a challenge we need to think about at some point with a management tool - how do you make it reliable, can you cluster them etc.

[preardon]

I like the idea of a "Management" Host, it is sort of what I am doing internally at work for brighter

[iancooper]

@preardon I think it's a useful pattern.

[preardon]

I've been thinking about this, the way I see it there should be 2 actions:

• Archive
• Remove

Remove is of course the easy one, however I have a question around Archive, the very first thing I think of is archiving into the same store (i.e. there is a MsSql Outbox and a MsSql OutBox Archive), however what if instead we preferred to archive to Blob instead, this would of course mean that we would need to create "Outbox Archivers" / "Messaging Archives"

@iancooper if you're not opposed to this, I can get an ADR around this and then maybe give it ago

[iancooper]

@preardon What do we think the status of this is?

[preardon]

@iancooper looks like I completed it and updated the documentation quite a while ago, I'll close it now