Feature request: Queues input trigger

[erkkikeranen]

To integrate loosely-coupled components together, it would be nice that we could use Queue service provided by cloud provider.

Queues are required because they keep the state of delivery and can mark messages that could not be delivered. The clients can be disconnected and messages delivered later when the clients become online.

There are some drawbacks in Queue implementation in AWS, because SQS must be constantly polled to receive new messages (which is awkward in the serverless context).

SNS can be used to implement sort of a Queue that can be pushed to Lambdas.

In Azure there are two possible Queue solutions that have a very similair abstraction. It would be nice if user could choose which implementation would be used.

AWS could use SNS with the lambda as a subscriber.

AWS SNS considerations compared to Azure:

• not FIFO
• no duplicate detection (at-least-once delivery)
• no poison / dead-letter queue (notifications unable to deliver in 13h are deleted)

Azure available solutions:

• Service Bus Queue
• Storage Queue

Note: Service Bus Topic has similair abstraction like AWS SNS, but currently if Azure function runtime on javascript fails during processing of topic messages, immediate retries will occur and messages are never marked failed and moved to dead-letter queue (they fail for the amount of allowed retries and then 'succeed') so information of failed delivery is hard to backtrack.

The Azure Queue polling is built in to the serverless runtime, which is an advantage so the experience is that new messages from the queue are 'pushed' to the handler.

Developer is still responsible for dealing with issues like:

• handling of duplicates (if issue)
• handle order of handling because it might not be correct when several messages are scaled out to multiple functions instances (if issue)

hedge.edn

{:queue {:handler name
         :queue queue-name
         :connection connection-string}}

:queue is the name of the queue :connection (you define in Azure which env variable has the auth&connection setting to the queue)

handler

(defn handler [message]
    (info "The received message from queue: " message))

You could also use a different designator in hedge.edn when specifying a topic (be it aws, azure or something else) (in case you end up using azure servicebus topic which is similar to SNS but has drawbacks in this context).

{:queue {:handler name
         :queue topic-name (sns topic or azure service bus topic)
         :subscription topic-subscription-name (if omitted = topic-name-subscription)
         :connection connection-string}}

Note on subscription-name for Azure: created if does not exist under topic

Depending on underlying queue and platform used, the queue-format would abstract to a default topic & subscription combination and vice versa if given as topic & subscription a queue would use the topic as it's name.

In Azure, default rights for sbqueue/sbtopic access is Manage if not given. I don't know if makes sense to add Listen option, comparing to AWS.

• [x] Azure implementation
• [x] AWS implementation
• [x] Documentation

[jikuja]

Also: https://hackernoon.com/applying-the-pub-sub-and-push-pull-messaging-patterns-with-aws-lambda-73d5ee346faa

[erkkikeranen]

removed cloud configuration overloading to other issue