Allow to send and receive content that is not base64 encoded

[danielmarbach]

More context https://discuss.particular.net/t/help-getting-amazonsqs-transport-native-integration-sample-running-with-json-serialization-on-nsb-6/3069

Originally, the community transport did base64 encode the message body to circumvent the character limitations of SQS. This encoding was added to be "helpful" and was kept to stay backward compliant with the community edition. The downsides of base64 encoding is that it significantly increases the message size, increases the allocations when sending and receiving messages and also makes it hard to do native integration with Lambda since Lambda will send JSON payloads as is without encoding it in base64.

We need to figure out whether we can discover that a message is base64 encoded or not when receiving, and handle those scenarios gracefully somehow.

[mgmccarthy]

@danielmarbach do you need to discover if a message is base64 encoded or allow some type of endpoint config that a user could specify NOT to use base64 encoding/decoding?

In my case, I know I don't need base64 encoding for the native integration, so I don't need the endpoint to figure that out for me, I need to tell the endpoint not to base64 decode incoming messages

Something like:

var transport = endpointConfiguration.UseTransport<SqsTransport>();
transport.DisableBase64Encoding()
transport.DisableBase64Decoding()

Control both the encoding/decoding?

[danielmarbach]

@mgmccarthy yep that would definitely also work. I was only brain dumping things into the issue. Thanks for picking it up and thinking about it even further.

[natilivni]

We would be very interested in this as well. The base 64 encoding makes it difficult to observe the contents of messages when debugging.

[mitchell-etter]

In addition to the body being base-64 encoded, another difference in NServiceBus behavior with AWS SNS/SQS is that the message headers are serialized into the SNS message body instead of being added to the message attributes. The only SNS message attribute is NServiceBus.MessageId. It would be nice if SNS message filtering could be used with NServiceBus messages.

Sample SNS message body:

{
  "Headers": {
    "NServiceBus.MessageId": "09d7bc34-436f-48c8-82f1-af0a011e9594",
    "NServiceBus.MessageIntent": "Publish",
    "NServiceBus.ConversationId": "3c701e02-202f-46fa-892f-af0a011e9594",
    "NServiceBus.CorrelationId": "09d7bc34-436f-48c8-82f1-af0a011e9594",
    "NServiceBus.OriginatingMachine": "MyComputer",
    "NServiceBus.OriginatingEndpoint": "NsbSnsProducer",
    "NServiceBus.ContentType": "application/json",
    "NServiceBus.EnclosedMessageTypes": "NsbSnsProducer.Messages.MessagePublished, NsbSnsProducer.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
    "NServiceBus.Version": "7.7.4",
    "NServiceBus.TimeSent": "2022-09-08 17:23:25:287782 Z"
  },
  "Body": "77u/eyJUZXh0IjoiU29tZSBkZWZhdWx0IHRleHQifQ==",
  "S3BodyKey": null
}

[timbussmann]

Another case where the current base64 encoding makes external integration scenarios more difficult: https://discuss.particular.net/t/aws-ses-integration-with-native-messages/3167

[DavidBoike]

Talked with a user and the problem is that a native message through SNS might look something like this (lifted out of a screenshot with client-specific data redacted - copy/paste fidelity might not be 100%):

{
  "Type": "Notification",
  "Messageld": "8843c065-09d1-588e-87cb-000000000000",
  "TopicArn": "arn:aws:sns:us-east-1:000000000000:sns-topic-name",
  "Subject": "Amazon SES Email Receipt Notification",
  "Message": "…", // Could be a single value or could be a JSON structure
  "Timestamp":"2022-09-09712:07:18.5602",
  "SignatureVersion": "1",
  "Signature": "DpRRisJk8TW1WZ5g0/n8QDzkxkw00IRe4g1j5DvkEauLdsK9kIgwZ4pVSLEC2/24Pjd3NLflUtMXEqlLGfVBcQHvz5nXyycHen6Qwq/Sg72FTx9n0N6Vzph6yuG91vOcolg1wodd"
  "SigningCertURL": "https://sns.us-east-l.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
  "UnsubscribeURL": "https://sns.us-east-l.amazonaws.com/?Action-Unsubscribe&SubscriptionArn-arn:aws:sns:us-east-1:000000000000:local-name"
}

This can definitely not be deserialized to a NServiceBus.Transport.SQS.TransportMessage, the structure of which can be seen in this comment from @mitchell-etter.

Here is the code involved in decoding the contents of the Amazon.SQS.Model.Message type into a NServiceBus.Transport.SQS.TransportMessage, and unfortunately it affords no opportunity for non-NServiceBus code to have an opinion on how that should be done. This activity happens before the pipeline is invoked so there's no opportunity to "fix it" with a pipeline behavior either.

That block of code also assumes that a MessageTypeFullName attribute being availble means we're in a "native integration" mode, but that means native integration is only possible if the sender is explicitly tailoring the message for an NServiceBus receiver…which isn't really "native" integration.

Switching ever so briefly into solution mode, it seems that the true contents of an SQS message could be so many different things that the end user should not be locked out of the process of converting Amazon.SQS.Model.Message to NServiceBus.Transport.SQS.TransportMessage. Even if we were to detect and handle more situations "out of the box" the user should still be able to influence this process by providing a Func<Amazon.SQS.Model.Message, NServiceBus.Transport.SQS.TransportMessage> … except NServiceBus.Transport.SQS.TransportMessage is an internal class.

[jdaigle]

@DavidBoike how open are you to community contributions for building something that could allow a user to convert Amazon.SQS.Model.Message to NServiceBus.Transport.SQS.TransportMessage?

I'm working on a project to migrate from a home-grown "service bus" built on SQS to using NServiceBus, and it would be great to be able to "adapt" the existing messages sent to the queue to what NServiceBus expects (in addition to other native integration scenarios like SNS, etc).

Maybe it really is as simple as a new configuration option, a simple func like you described. Obviously with some under-the-hood changes, and possibly making TransportMessage public.

[DavidBoike]

@jdaigle I wish I could give you a definitive answer. I'm honestly not now sure that a simple func like I described would even be a good idea. The idea of making TransportMessage public would certainly give us pause.

What might be necessary is for more of a V2 message representation, where headers would be stored in the SQS message's Attributes dictionary and the transport in general is more forgiving. This would likely be a multi-step process: First make the transport understand and process both V1 and V2 message representations, while you make it an opt-in feature for the transport to send messages in the V2 format to preserve wire compatibility with un-upgraded destination endpoints. Then in another phase (likely a major version) the V2 representation would become the default.

We have an AWS enhancement slated for the near future and it's possible (but not guaranteed) they could take up this issue. I certainly wouldn't want you to submit a PR and then have us reject it because it does not comport with the direction we want to go.

[jpalac]

This has been addressed in release 6.1.0