Expose DLQ URL

[pano-skylakis]

I'm currently retrieving the DLQ URL by referencing the logical ID of the DLQ that is generated after the first deployment.

For example in my serverless.yml:

environment:
  QUEUE_URL: ${construct:queue.queueUrl}
  DLQ_URL: !Ref queueDlqAB1234C5

Ideally, this value is exposed on the queue when referencing the construct, so it could look like the following:

environment:
  QUEUE_URL: ${construct:queue.queueUrl}
  DLQ_URL: ${construct:queue.dlqUrl}

I've had a bit of a dig around and I think we can possibly achieve this by changing the following in Queue.ts

variables(): Record<string, unknown> {
        return {
            queueUrl: this.queue.queueUrl,
            dlqUrl: dlq.queueUrl,
            queueArn: this.queue.queueArn,
        };
    }

[t-richard]

Hi! Thanks for opening this issue.

In Lift, we like to start discussions from the use case (the why) instead of the solution (the how) ?

Could you describe the problem you're trying to solve with this ?

PS: it was already asked in #63 but the use case was different from what is envisioned for the queue construct and led to a totally different discussion.

[pano-skylakis]

Thanks for the response!

For sure, I'll elaborate.

In order to get access to the DLQ URL, I have to first deploy the queue and then either get the logical ID of the DLQ from the generated cloudformation template, or get the DLQ URL from the AWS console.

We end up with two different methods for accessing the queue URL and setting it as an environment variable, and getting the DLQ URL involves additional steps.

The reason for needing the DLQ URL is to manually push messages to the DLQ when an error occurs which we know will not resolve on a retry.

Edit: Just as a note, I'm also manually configuring the DLQ policy with the SQS:SendMessage action 🤔

[mnapoli]

The reason for needing the DLQ URL is to manually push messages to the DLQ when an error occurs which we know will not resolve on a retry.

Ohh interesting 🤔 That makes sense, even though I never heard that use case before. I'll ask around, but that sounds reasonable to me? What do others think?

[t-richard]

The reason for needing the DLQ URL is to manually push messages to the DLQ when an error occurs which we know will not resolve on a retry.

I'm divided on this. Maybe it makes sense in this specific domain use case you are facing but here's my two cents.

I use retry and DLQs to make sure my code is retried if a recoverable error is happening : a third party is down, rate limiting/throttling is applied to my API calls, network errors, etc If it fails several times and goes to the DLQ, it probably means that it was not recoverable so either there was a huge outage OR I made a mistake in my code/config/infrastructure and a case is not handled properly. In either case, those are unknown ways of failure that I want to be able to manually recover from by moving messages again to the main queue when fixed.

If you are able to say that an error will not be recoverable, it's a known failure mode and it probably means that the message should be dropped or should be handled by another dedicated system (posting a new message in a different SQS queue, Eventbridge, etc). I don't see why you would send it to the DLQ in that case as you'll never put it back into the main queue for processing because it would still be failing for known reasons.

I just can't think of a case where that would be useful but please prove me wrong if you think I am :smiley: :pray:

[benny-bp]

In living in an event-driven world, i'd like to be able to hook a function to the DLQ and consumer the messages and send notifications once a message hits it while simultaneously inserting the faulty message in a separate table where i can review the payload. The latter is of course a specific case, but i'd definitely want to hook up a function that can perform arbitrary tasks once a messages hits the DLQ.

[jag-ermeister]

I would also like to see the DLQ url exposed. @t-richard you might be interested in the use-case I am running into. We basically want to set up several queues (with attached DLQs). As we know, the messages in the DLQ will expire and disappear after 14 days. So that we don't lose those, we want to build a lambda function handler that we will attach to our DLQs, and will grab those messages and send them to Slack where we have better visibility to we can triage those issues.

In addition to that, I simply don't see any negative consequence in exposing the DLQ url, so why not offer the flexibility to the consumer?