Unable to use dead letter queue in serverless-offline-sqs

[esetnik]

@godu Since the latest update I am no longer able to use autoCreate: true with dead letter queue configuration. This was working previously.

I see the following error:

Issues checking in progress...
offline: AWS.SimpleQueueService.NonExistentQueue: AWS.SimpleQueueService.NonExistentQueue; see the SQS docs.
    at Request.extractError (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at Request.emit (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at callNextListener (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
    at IncomingMessage.onEnd (/app/node_modules/serverless-offline-sqs/node_modules/aws-sdk/lib/event_listeners.js:307:13)
    at IncomingMessage.emit (events.js:327:22)
    at IncomingMessage.emit (domain.js:482:12)
    at endReadableNT (_stream_readable.js:1221:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

custom:
  serverless-offline-sqs:
    autoCreate: true
    apiVersion: '2012-11-05'
    endpoint: http://localstack:4576
    region: us-east-1
    accessKeyId: root
    secretAccessKey: root
    skipCacheInvalidation: false
resources:
  Resources:
    EmailQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: email-${self:provider.stage}
        MessageRetentionPeriod: 1209600
        RedrivePolicy:
          deadLetterTargetArn:
            'Fn::GetAtt':
              - EmailDeadLetterQueue
              - Arn
          maxReceiveCount: 5
    EmailDeadLetterQueue:
      Type: 'AWS::SQS::Queue'
      Properties:
        QueueName: email-dlq-${self:provider.stage}
        MessageRetentionPeriod: 1209600

[dvejmz]

Same issue here on [email protected] with similar config.

[YOU54F]

Looks like the RedrivePolicy is ignored, I was looking into it and found this plugin which I have got working

https://www.npmjs.com/package/serverless-offline-sqs-dlq

[Uliversal]

I'm using a similar setup as @esetnik Seems it's searching for a kinesis queue, not a sqs queue? I got this error: offline: AWS.SimpleQueueService.NonExistentQueue: Could not find DLQ for arn:aws:kinesis:eu-central-1:000000000000:voucher-service-user-confirmed-dlq-local

If I replace "kinesis" with "sqs" in the code it works perfectly:

          switch (type) {
            case 'AWS::SQS::Queue': {
              const queueName = get([resourceName, 'Properties', 'QueueName'], Resources);
              return [
                key,
                `arn:aws:kinesis:${this.options.region}:${this.options.accountId}:${queueName}`
              ];
            }
            default: {
              return null;
            }
          }

[will-holley]

@YOU54F is correct; the NonExistentQueue is in reference to a non-existent EmailDeadLetterQueue and not EmailQueue. Creation should ordered as a dependency-DAG, but dependencies are not taken into consideration because the serverless.yml function – rather than resource – declarations are parsed to enqueue queue creation.

[araujoigor]

Facing the same problem.

I love this project, and to be honest I haven't looked into the details of how it is built, but I am curious to understand why the autoCreate flag does not scan all of the AWS::SQS::Queue resources and create them automatically.

I guess this would be a nice to have feature request and a even better behavior from the UX standpoint.

[will-holley]

@araujoigor, I went ahead and solved this (my) use case here.

[AdrieanKhisbe]

@Uliversal suggested fix as been incorporated in #148 via 178b52c. Thanks for highlighting this regression

@esetnik and others, [email protected] has been released and contain the fix. Please tell if it solves issue for you

[araujoigor]

@AdrieanKhisbe Thanks! It works, yes.

But can you consider my suggestion of scanning the CloudFormation resources and automatically creating them instead of only creating those linked to a lambda function?

I am suggesting this because in order for this to work at the moment, you need to link the Dead Letter queue to a Lambda event array. But not always this is what the user wants to do. In my case, for instance, I need to change my serverless.yml file everytime I want to run it locally (linking the DL queue to one of my lambdas).

I would be happy to help developing such a feature, if this gets accepted.

[AdrieanKhisbe]

@araujoigor not sure I have much time for it, but first of all I might need to have a look to have an idea why that partially works for you.

Initial issue from @esetnik was about a regression that has just been fixed. So far autoCreate is only for queue registered as event. cf autocreate location sqs.js#L65

Might be a good idea to open a dedicated issue on that subject :)

[marcusstenbeck]

I'm having the same issue. I have a deadletter queue that's not linked to any lambda handlers. Very annoying.

If it's pretty far out then I suggest adding to the readme that autoCreate only creates queues that are linked to a lambda function.

[renatoargh]

Hello, any progress on this issue?

[renatoargh]

Just wanted to share my work around;

So I created a docker-compose file to spinup elasticmq, and this way I can run an initialization script that creates the DLQ before I run sls offline start. My work around solution looks like this:

---
version: '3'
services:
  elasticmq:
    image: softwaremill/elasticmq-native
    container_name: elasticmq
    ports:
      - "9324:9324"
      - "9325:9325"

  elasticmq-init:
    image: amazon/aws-cli
    depends_on:
      - elasticmq
    restart: "no"
    environment:
      AWS_ACCESS_KEY_ID : FAKEKEYID
      AWS_SECRET_ACCESS_KEY: FAKEACCESSKEY 
    entrypoint: [
      "aws", 
      "sqs", 
      "create-queue", 
      "--queue-name", 
      "response-queue-dlq.fifo", 
      "--endpoint-url", 
      "http://elasticmq:9324",
      "--region",
      "us-east-1"
    ]

then on package json I have the following startup script:

{
  "scripts": {
    "offline-start": "docker-compose up -d && sls offline start"
  }
}