NServiceBus
NServiceBus copied to clipboard
Particular
Allow the endpoint to use different DataBus types when forwarding to audit or error Qs
In a system the DataBus implementation is shared across all the endpoints that needs to talk to each other and is also used by the error and audit features.
The issue when sharing the chosen DataBus between endpoint normal operations and error and audit management is that these 3 concepts have very different cleanup policies:
- normal endpoint operations: when the message is processed
audit: as the message expires if the audit si done via SC or when Ops decideerror: potentially never as an error message can be retried at any point in time or archived but never deleted
/cc @ramonsmits
ramon [5:51 PM] @here : When a message is read from the audit queue by ServiceControl, does it store the blobs from the databus in the message?
andreas [5:52 PM] no it does not, pretty sure we just ignore the databus properties
ramon [5:52 PM] @andreas: So its stores the wire message body?
andreas [5:52 PM] yes
[5:52] so databus data is gone
mike [5:53 PM] It doesn't even get deserialized.
[5:53](kind of). If it's not binary we convert it into a string
andreas [5:53 PM] huh, we're not reading the data part of databus props?
mike [5:56 PM] The body we get over the wire is pushed into Body Storage as a binary stream. IF it's non-binary we also convert that stream into a string and add it to the Raven Document for full-text search. If databus needs special treatment, it's not there
andreas [5:57 PM] yea, databus property content needs to be read from the configured databus and we don't do that (SC doesn't have any settings to configure a databus)
mike [5:57 PM] It also doesn't necessarily have a configured deserializer (although multi-deserializers goes a way to resolving that).
andreas [5:58 PM] but databus is one more level of indirection
mike [5:58 PM] And we don't have the class definition of the message type so any attributes are lost
andreas [5:58 PM] http://docs.particular.net/nservicebus/messaging/databus DataBus feature
mike [5:58 PM] Can we even tell it's a databus property?
andreas [5:58 PM] yes we can
[5:58] and there is also headers to tell
[5:59] not convicened that we need to store the actual databus content
mike [5:59 PM] And given that the data is already in the databus do we need to ingest it?
andreas [5:59 PM] since that's most likely large binary stuff
mike [5:59 PM] Or should SC tell the Databus what to do with it?
[5:59] i.e. Should we have an SC-Controlled-Expiration-Policy for DataBus
andreas [6:00 PM] I'd say we just ignore databus props like we do today
[6:00] perhaps we can flag messages as containing databus properties in SI or something
mike [6:00 PM] That's problematic. If the Databus data expires and then you retry the message
[6:01] Although to be fair, it's no different than if the endpoint was down for a bit and the databus expires the message
ramon [6:02 PM] @mike @andreas Is this really a ServiceControl concern?
[6:02] Shouldn't the databus provider be responsible for purging and audit hooks?
@mauroservienti since audit and error involves SC like you mentioned would it make sense to move this issue to platform development and flesh it out there?
moved to https://github.com/Particular/PlatformDevelopment/issues/879
The issue was resolved for the Azure Databus implementation, however, the problem persists in the FileShareDatabus. Re-opening this issue.
