Cross-account communication in SQS transport

[mauroservienti]

Describe the feature.

Context

Related issues:

• https://github.com/Particular/NServiceBus.AmazonSQS/issues/1654
• https://github.com/Particular/NServiceBus.AmazonSQS/issues/496
• https://github.com/Particular/NServiceBus.AmazonSQS/issues/504

We have conducted short research, and it seems like most people tend to lean towards not treating ARN as secrets but more like known service URLs. There are, however, some caveats.

• knowing the ARN allows one to attempt to send commands to a resource. By default, these will fail because they will be done outside the resource security context but would pose a threat if resource access rights are misconfigured. This message-loss scenario can be mitigated by configuring SNS topics to report automatically delivery failures in CloudWatch (AmazonSQS issue #1676)
• the ARN includes the account number, which does not pose any risks at the moment but might be of some use to someone who plans an attack (not necessarily on the actual leaked ARN). To mitigate any security/privacy-related concern, the destination ARN to route to could be stored outside the endpoint configuration code in a vault. We could enable users to encrypt header values like they can with property values. At that point, the value traveling on the wire is no longer exposed.

In other words, exposing ARN values in message headers seems OK. It would, however, require changing the addressing scheme of SQS. Currently, the SQS transport address consists of a single value, either a queue name or a prefixed queue name. These two forms cannot be distinguished as there is no delimiter between the prefix and the name.

The SQS transport uses the queue name as the transport address (TODO: is that defined?), which is the address that is exchanged between endpoints in the headers i.e.

• NServiceBus.ReplyToAddress
• NServiceBus.FailedQ
• NServiceBus.SubscriberAddress

This transport address is built by concatenating the configured queue prefix with the queue name (derived from the endpoint name). The concatenation method can be customized by providing a queue name generator function. Due to the lack of representation of the queue prefix as a separate thing in the transport address, there is no way to distinguish whether a given transport address is prefixed. For that reason, the queue name generator function used in IMessageDispatcher needs to be idempotent, i.e., apply the prefix only if it appears that it has not been applied yet.

Problem

To support cross-account communication (which is only a security configuration concern), the transport address in SQS needs to be changed to include the account ID. The challenge is that existing legacy endpoints running SQS cannot be expected to be upgraded to communicate with endpoints running the new version of the transport.

In addition to that, could the new form of the transport address solve the problem of requiring an idempotent queue name generator?

Assumptions

The transport address is only transmitted on the wire in the headers

The headers that contain the transport address contain only that address and no other value (nor additional whitespace)

The only transport address that is being written to the headers is the address of the endpoint that is sending the message

The transport can add a header to an outgoing message

The transport can substitute a header value in the incoming message

Current account is available

The account ID for the configured credentials can be determined by the transport.

Confirmed by remark in the documentation of the GetQueueUrl API:

To access a queue that belongs to another AWS account, use the <code>QueueOwnerAWSAccountId</code>
parameter to specify the account ID of the queue's owner. The queue's owner must grant
you permission to access the queue. For more information about shared queue access,
see <code> <a>AddPermission</a> </code> or see <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-writing-an-sqs-policy.html#write-messages-to-shared-queue">Allow
Developers to Write Messages to a Shared Queue</a> in the <i>Amazon SQS Developer

Address formats

The ARN arn:partition:service:region:account-id:resource-id is the new canonical format of the transport address (a format returned by ToTransportAddress). Two other supported transport address formats are:

• just the resource-id part prefixed with : e.g., :my-queue. This means a queue in the current credentials account/region with a given name. No prefix is applied to it.
• just the queue name, e.g., my-queue. This means a queue in the current credentials account/region with a name that results from applying the configured prefix (via QueueNamePrefix API) using the queue name generator function (customizable).

The :resource-id form is never present on the wire. When provided in the endpoint's configuration, it is always expanded to full ARN.

Configuration APIs

Based on the above, the ToTransportAddress method has to return the canonical format, the ARN. It needs information about the ACCOUNT and REGION for the queue to do so. That information could be managed by the transport, but the more canonical way of doing this is by taking advantage of the instance mapping feature the same way as MSMQ does with the machine names. This way the data may come from various sources, including centralized configuration. The transport should, however, provide some API for setting, similar to how SQLT does it, it e.g.

sqs.UseRegionForEndpoint("MyEndpoint", "us-east-2");
sqs.UseAccountForEndpoint("MyEndpoint", "123456789");

Considering that the de facto standard in AWS is to use environment variables to configure services and resources, it would be good if the transport allows configuring the region and the account for the endpoints to route to via environment variables.

Note that SQLT provides similar methods for queue names. These are used for APIs that accept transport addresses, such as SendFailedMessagesTo but are redundant because their APIs already accept the canonical address form (ARN). In the SQL Transport, these APIs are kept for compatibility reasons, and there is no good reason to add them to SQS.

Solution

On the sending side, the solution consists of the following steps:

• Detect if the outgoing message headers contain a NServiceBus.Transport.Sqs.ARN header
  • If so, parse the header according to the <header_name_1>=<header_value_1>;<header_name_2>=<header_value_2> format (escaping the = and ; signs with == and ;;) and store in a dictionary
• In the IMessageDispatcher, scan all message headers and detect ones whose value is equal to the transport address of the local endpoint (main receiver's ReceiveAddress)
• If there are any such headers found, add each of them to the parsed dictionary as a key/value pair
• Serialize the dictionary back to the wire format (semicolon-separated key/value pairs)

On the receiving side, the solution consists of the following steps:

• In the InputQueuePump makes a copy of the incoming message headers
• Detect if the incoming message contains the NServiceBus.Transport.Sqs.ARN header. If so, parse it to a dictionary
• For each entry in the dictionary, substitute a message header with the matching name with the value from the dictionary
• Push the modified message to the pipeline

As a result, the updated endpoints would be able to participate in cross-account communication while the non-updated endpoints will still be able to talk to updated endpoints within the same account.

Note that the header value substitution must happen before the message is handed over to the pipeline because various pipeline components may use or even store (sagas) the affected header values (sagas).

Note: it is not enough to have a single ARN value for the entire message as different substitution values can be added during the lifetime of a message e.g.

• An endpoint sends a subscribe message with NServiceBus.Transport.Sqs.ARN value of NServiceBus.SubscriberAddress=<subscriber ARN>.
• A publisher endpoint fails to process the subscription and forwards that message to the error queue with NServiceBus.Transport.Sqs.ARN value of NServiceBus.SubscriberAddress=<subscriber-ARN>;NServiceBus.FailedQ=<publisher-ARN>.

Scenarios

Each scenario is considered in three cases, depending on the version of the participating endpoints. New endpoint means one that understands the new address format, and Old means one that doesn't.

Hybrid mode message-driven pub-sub

In this mode, a subscribe message is sent to the publisher endpoint to subscribe to a given event. The destination of the subscribe message is configured in the routing configuration using the RegisterPublisher API.

config.ConfigureRouting().EnableMessageDrivenPubSubCompatibilityMode().RegisterPublisher(typeof(MyEvent), "MyPublisher");

New to New

1. Additional API is be used to specify region and account: sqs.UseAccountForEndpoint("MyPublisher", "ACCOUNT2");

2. The subscribe message contains two headers:

• [NServiceBus.SubscriberAddress: MySubscriber]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.SubscriberAddress=arn:aws:sqs:REGION:ACCOUNT:MySubscriber]

1. The subscribe message is sent to the queue arn:aws:sqs:REGION:ACCOUNT2:MyPublisher as the instance mapping feature determines.
2. The message pump of the new endpoint replaces the value of the NServiceBus.SubscriberAddress header with the corresponding value from the NServiceBus.Transport.Sqs.ARN header (arn:aws:sqs:REGION:ACCOUNT:MySubscriber) and passes the message to the processing pipeline.
3. The hybrid mode behavior picks up the value arn:aws:sqs:REGION:ACCOUNT:MySubscriber from the well-known header NServiceBus.SubscriberAddress and stores in the subscription store
4. When an event is published, the message is sent to the queue identified by arn:aws:sqs:REGION:ACCOUNT:MySubscriber even though that queue is not defined under the account running the publisher (ACCOUNT2)

Note: in case the subscription already existed before the publisher has been upgraded, a second entry is going to be created in the subscription store as the store address (arn:aws:sqs:REGION:ACCOUNT:MySubscriber) is different from the existing one (MySubscriber). This is not going to create any issues if both endpoints are in the same account (which is true since the original subscription exists) because both subscription entries would contain the same endpoint name (MySubscriber), and the publisher routing logic would round-robin between the entries.

New to Old

1. The subscribe message contains two headers:

• [NServiceBus.SubscriberAddress: MySubscriber]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.SubscriberAddress=arn:aws:sqs:REGION:ACCOUNT:MySubscriber]

1. The subscribe message is sent to the queue arn:aws:sqs:REGION:ACCOUNT:MyPublisher as determined by the instance mapping feature (no mapping exists, so the current credential's account is used)
2. The message pump of the old endpoint passes the message as-is to the processing pipeline.
3. The hybrid mode behavior picks up the value MySubscriber from the well-known header NServiceBus.SubscriberAddress and stores it in the subscription store. Note that the additional header ARN.NServiceBus.SubscriberAddress is ignored.
4. When an event is published, the message is sent to the queue identified by the result of calling GetQueueUrl on the value MySubscriber.

Note that in this scenario, the subscriber (new) cannot use the UseAccountForEndpoint API to subscribe for events published by a publisher using a different account because, even though that subscriber would receive the subscribe message, it would not understand the ARN header and would attempt to publish its events to a queue inside its own account.

Old to New

1. The subscribe message contains only one header:

• [NServiceBus.SubscriberAddress: MySubscriber]

1. The subscribe message is sent to the queue identified by the result of calling GetQueueUrl on the result of QueueNameGenerator(QueueNamePrefix, ToTransportAddress(subscriberEndpoint)).
2. The message pump of the new endpoint passes the message as-is to the processing pipeline.
3. The hybrid mode behavior picks up the value MySubscriber from the well-known header NServiceBus.SubscriberAddress and stores it in the subscription store
4. When an event is published, the message is sent to the queue identified by the result of calling QueueNameGenerator(QueueNamePrefix, "MySubscriber") because the value picked up from the subscription store is in the legacy format.

Send and Reply

Suppose there are two endpoints, Sender and Receiver. The Sender is configured to route messages of type MyRequest to the Receiver.

config.ConfigureRouting().RouteToEndpoint(typeof(MyRequest), "Receiver");

The Receiver endpoint does not have any routing configuration. The message MyReply is routed back to the Sender based on the NServiceBus.ReplyToAddress header.

New to New

1. Additional API is be used to specify region and account: sqs.UseAccountForEndpoint("Receiver", "ACCOUNT2");

2. The request message contains two headers:

• [NServiceBus.ReplyToAddress: Sender]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:Sender]

1. The request message is sent to the queue arn:aws:sqs:REGION:ACCOUNT2:Receiver as the instance mapping feature determines.
2. The message pump of the new endpoint replaces the value of the NServiceBus.ReplyToAddress header with the corresponding value from the NServiceBus.Transport.Sqs.ARN header (arn:aws:sqs:REGION:ACCOUNT:Sender) and passes the message to the processing pipeline.
3. The message handler calls the context.Reply API to send the MyReply message. The available in the context reply header (arn:aws:sqs:REGION:ACCOUNT:Sender) is used as a destination for the new message.
4. The reply message is received by the message pump of the sender.

New to Old

1. The request message contains two headers:

• [NServiceBus.ReplyToAddress: Sender]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:Sender]

1. The subscribe message is sent to the queue arn:aws:sqs:REGION:ACCOUNT:Receiver as determined by the instance mapping feature (no mapping exists, so the current credential's account is used)
2. The message pump of the old endpoint passes the message as-is to the processing pipeline.
3. The message handler calls the context.Reply API to send the MyReply message. The available reply header (Sender) in the context is used as a destination for the new message.
4. The reply message is received by the message pump of the sender.

Note that in this scenario, the sender (new) cannot use the UseAccountForEndpoint API to send messages to a different account because, even though that receiver would receive the request message, it would not understand the ARN header and would attempt to send the reply to a queue inside its own account.

Old to New

1. The request message contains only one header:

• [NServiceBus.ReplyToAddress: Sender]

1. The request message is sent to the queue identified by the result of calling GetQueueUrl on the result of QueueNameGenerator(QueueNamePrefix, ToTransportAddress(subscriberEndpoint)).
2. The message pump of the new endpoint passes the message as-is to the processing pipeline.
3. The message handler calls the context.Reply API to send the MyReply message. The available reply header (Sender) in the context is used as a destination for the new message.
4. The reply message is received by the message pump of the sender.

Bridge Send and Reply

The bridge moves messages between transports by creating shadow queues. If there is an endpoint MyEndpoint on one side of the bridge, the bridge assumes there is a shadow queue whose name is derived from the name MyEndpoint on the other side. Messages from the shadow queue are being received by the bridge and forwarded to the destination queue on the other side.

For the bidirectional send-reply communication to work, both endpoints must be registered with the bridge on their respective sides. The transport bridge substitutes the NServiceBus.ReplyToAddress header with the value appropriate for the transport on the other side.

This section assumes that the bridge is always upgraded before any endpoints, so we don't need to analyze scenarios with the old bridge and new endpoint. Scenarios involving the new bridge and the old transport work exactly like in the current version, so they don't need to be analyzed either. The only changes happen when both the bridge and the endpoint use the ARN-aware versions of the SQS transport.

Sending from SQS

Suppose there is an SQL transport endpoint Receiver and SQS transport endpoint Sender. The Receiver is deployed to catalog nservicebus and default schema dbo, so its transport address is Receiver@dbo@nservicebus.

1. The bridge is configured in the following way:

• sqs.HasEndpoint("Sender", "arn:aws:sqs:REGION:ACCOUNT:Sender")
• sqlt.HasEndpoint("Receiver", "Receiver@dbo@nservicebus")

1. This configuration results in the endpoint registry containing the targetEndpointAddressMappings with following values (based on this code):

• ["arn:aws:sqs:REGION:ACCOUNT:Sender": "Sender"]
• ["Receiver@dbo@nservicebus": "Receiver"]

1. The message contains the following headers:

• [NServiceBus.ReplyToAddress: Sender]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:Sender]

1. The message pump on the SQS side of the bridge receives the message and replaces the NServiceBus.ReplyToAddress header value with the ARN from the NServiceBus.Transport.Sqs.ARN header
2. The bridge shovel extracts that value and passes it to the endpoint registry for translation.
3. The translation results in the value Sender, which is put into the NServiceBus.ReplyToAddress header in the message on the SQL Server transport side.
4. A reply is being sent by the receiver to the Sender shadow queue managed by the bridge
5. The bridge shovel extracts the NServiceBus.ReplyToAddress header value (Receiver@dbo@nservicebus) and passes it to the endpoint registry for translation.
6. The translation results in Receiver
7. The reply is forwarded to the arn:aws:sqs:REGION:ACCOUNT:Sender as the TargetEndpointDispatcher is aware of the ARN queue address configured in the HasEndpoint API.

Receiving in SQS

Suppose there is an SQL transport endpoint Sender and SQS transport endpoint Receiver. The Sender is deployed to catalog nservicebus and default schema dbo, so its transport address is Sender@dbo@nservicebus.

1. The bridge is configured in the following way:

• sqlt.HasEndpoint("Sender", "Sender@dbo@nservicebus")
• sqs.HasEndpoint("Receiver", "arn:aws:sqs:REGION:ACCOUNT:Receiver")

1. This configuration results in the endpoint registry containing the targetEndpointAddressMappings with following values (based on this code):

• ["arn:aws:sqs:REGION:ACCOUNT:Receiver": "Receiver"]
• ["Sender@dbo@nservicebus": "Sender"]

1. The message contains the following header:

• [NServiceBus.ReplyToAddress: Sender@dbo@nservicebus]

1. The bridge shovel on the SQL side extracts the NServiceBus.ReplyToAddress header value (Sender@dbo@nservicebus) and passes it to the endpoint registry for translation.
2. The translation results in Sender, and this value is put into the NServiceBus.ReplyToAddress header in the message on the SQS transport side.
3. A reply is being sent by the receiver to the Sender shadow queue managed by the bridge containing the following headers:

• [NServiceBus.ReplyToAddress: Receiver]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:Receiver]

1. The message pump on the SQS side of the bridge receives the message and replaces the NServiceBus.ReplyToAddress header value with the ARN from the NServiceBus.Transport.Sqs.ARN header
2. The bridge shovel extracts that value and passes it to the endpoint registry for translation.
3. The translation results in the value Receiver, which is put into the NServiceBus.ReplyToAddress header in the message on the SQL Server transport side.
4. The reply is delivered to the Sender@dbo@nservicebus queue.

ReplyToOriginator from Saga instance

New to New

The saga processing endpoint will see the NServiceBus.ReplyToAddress as an ARN and store it as such in the saga. When the ReplyToOriginator API is called, the message will be routed to that address.

New to Old

The saga processing endpoint will see the NServiceBus.ReplyToAddress in the legacy format (including prefix) and store it as such in the saga. When the ReplyToOriginator API is called, the message will be routed to that address.

Old to New

The saga processing endpoint will see the NServiceBus.ReplyToAddress in the legacy format (including prefix) and store it as such in the saga. When the ReplyToOriginator API is called, the message is going to be routed to the address determined the same way as described in the publish-subscribe scenario.

Sending a Failed message to ServiceControl and retrying it

This section assumes that the ServiceControl is always upgraded before any endpoints, so we don't need to analyze scenarios with old ServiceControl and new endpoints. Scenarios involving the new ServiceControl and the old transport work exactly like in the current version, so they don't need to be analyzed either. The only changes happen when both the ServiceControl and the endpoint use the ARN-aware versions of the SQS transport.

Suppose MyEndpoint is hosted in account ACCOUNT and ServiceControl is hosted in account ACCOUNT2. The endpoint is configured with a failed queue address as ARN:

cfg.SendFailedMessagesTo("arn:aws:sqs:REGION:ACCOUNT2:error");

1. The failed message contains the following headers:

• [NServiceBus.FailedQ: MyEndpoint]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.FailedQ=arn:aws:sqs:REGION:ACCOUNT:MyEndpoint]

1. The failed message is sent to the queue arn:aws:sqs:REGION:ACCOUNT2:error as configured.
2. The message pump of ServiceControl replaces the value of the NServiceBus.FailedQ header with the corresponding value from the NServiceBus.Transport.Sqs.ARN header (arn:aws:sqs:REGION:ACCOUNT:MyEndpoint) and passes the message to the failed message processing logic.
3. The FailedQ header is parsed to create an object representing the failure details.
4. The object mentioned above is used to create the header containing the target addres of the retry.
5. The staged retry message is forwarded using the target address header as the ultimate destination.

Sending Audit message to ServiceControl and replaying it

Currently not supported by ServiceControl

Message forwarding. i.e., Moved a handler to another endpoint

Forwarding creates an exact copy of the current message being processed and forwards it. If the forwarding endpoint is ARN-aware, the header collection of the incoming message may contain header values substituted with ARNs. As a result, given the incoming message with the following headers:

• [NServiceBus.ReplyToAddress: MyEndpoint]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:MyEndpoint]

the message forwarded to MyOtherEndpoint using ctx.ForwardCurrentMessageTo("MyOtherEndpoint") API would contain the following headers:

• [NServiceBus.ReplyToAddress: arn:aws:sqs:REGION:ACCOUNT:MyEndpoint]
• [NServiceBus.Transport.Sqs.ARN: NServiceBus.ReplyToAddress=arn:aws:sqs:REGION:ACCOUNT:MyEndpoint]

This means that an old endpoint, unaware of the ARN format, would be unable to process the message.

Gateway transfers

TODO: it should not be affected

Additional Context

No response