0.76.0: SNS -> publish to subscribed SQS queue, "Records" assumption leads to lost message, empty array.

[bitsofinfo]

using 0.76.0

My code is sending a message to SNS like this:

    sns = boto3.client(service_name='sns',
                                endpoint_url=http://localhost:4002)

    message = {
        "my_event": {
            "x": "something"
        }
    }

    # publish it
    response = sns.publish(
        TopicArn=event_topic_id,
        Subject="test-my-event",
        Message=json.dumps(message),
        MessageAttributes={
            "data_ingress_type": {
                "DataType": "String",
                "StringValue": "test"
            }
        })  

I have an SQS queue (in serverless-offline-sqs) that is subscribed to the event topic the code is publishing to (which is properly setup and I see the queue existing in elasticmq)

  serverless-offline-sns:
    port: 9324
    debug: true
    subscriptions:
      - topic: ${self:provider.environment.INGRESS_EVENT_TOPIC_NAME}
        queue: http://localhost:9324/queue/my-events

When I startup the app, I see the queue being successfully subscribed to my serverless-offline-sns created topic as expected.

When the code runs to send the SNS message, it is never received.

Tracking things down in the code I see that sns-server.publish() gets called and for each subscription it creates a new SNS message that ends up looking like this which wraps the message above that I published via the boto client:

https://github.com/mj1618/serverless-offline-sns/blob/6f6cde4525552dd2efd9f235fbbcf42ff5f47d76/src/sns-server.ts#L351

{
  "SignatureVersion": "1",
  "Timestamp": "2021-08-01T23:07:52.737Z",
  "Signature": "EXAMPLE",
  "SigningCertUrl": "EXAMPLE",
  "MessageId": "c0f69f87-ce52-483d-ac74-c61f579fee28",
  "Message": "{\"my_event\": { \"x\": \"something\"}}",
  "MessageAttributes": {
    "data_ingress_type": { "Type": "String", "Value": "test" }
  },
  "Type": "Notification",
  "UnsubscribeUrl": "EXAMPLE",
  "TopicArn": "arn:aws:sns:us-west-2:123456789012:my-events",
  "Subject": "test-my-event"
}   

The sns-server.publish() method then calls sns-server.publishSqs(). In particular this line erroneously assumes that that event contains a Records property array... which it doesn't... then yields an empty array [] ultimately resulting in my message being lost and never sent: https://github.com/mj1618/serverless-offline-sns/blob/6f6cde4525552dd2efd9f235fbbcf42ff5f47d76/src/sns-server.ts#L310

Seems like the sns-server.publishSqs() event is expecting the passed event to be in the format of the helpers.createSnsLambdaEvent() structure instead? https://github.com/mj1618/serverless-offline-sns/blob/6f6cde4525552dd2efd9f235fbbcf42ff5f47d76/src/helpers.ts#L75

Original issue comment: https://github.com/mj1618/serverless-offline-sns/pull/88#issuecomment-889958962

[erik-am]

For me it works if I specify it like this. (Using your example)

  serverless-offline-sns:
    port: 9324
    debug: true
    subscriptions:
      - topic:
          topicName: ${self:provider.environment.INGRESS_EVENT_TOPIC_NAME}
          rawMessageDelivery: 'true'
        queue: http://localhost:9324/queue/my-events    

Note the quotes in rawMessageDelivery: 'true'! It doesn't work as a boolean, it should be a string.

Then the line of code that you changed in your PR is not even touched, because it will take this branch:

https://github.com/mj1618/serverless-offline-sns/blob/5a29200a9861967151a24d81695e96576d2f674d/src/sns-server.ts#L348-L349

[rcambrj]

I made a similar fix to yours @bitsofinfo, but preserving the event shape for publishHttp. I also based from 0.74.0 due to #141 because later versions seem to be break on inclusion into typescript serverless templates.

This alone causes #143 for me. But instead of using the fix proposed by that same PR #143, I just disabled the AWS::SNS::Subscription resources based on an environment variable present only when using serverless-offline-*. This way, serverless-offline-sns does not create any subscriptions

With the serverless-offline-sns subscriptions disabled between SNS & SQS, I installed and configured serverless-offline-sqs+elasticmq and pointed serverless-offline-sns to elasticmq as described in #88.

That was quite a journey. Hope this helps some others.

[ormu5]

I also forked from a previous tag to get this working and am maintaining the code at https://github.com/ormu5/serverless-offline-sns.