New feature: aws-stepfunctionstasks-sqs

[knihit]

Step function has a callback pattern called 'Callback with Task Token' - https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html. It allows for slower back-ends to respond rather than getting throttled and fail.

###Use Case Most back-ends and AWS services have TPS limits may not be able to handle bursts or higher volume of streaming data. Having step function workflow, retries and data streams like MSK or Kinesis does reduce the throttling issues. But does not solve the problem where a step function task has a slower back-end and a successful completion of a task requires transitioning through all the states in the statemachine.

###Proposed Solution To use the callback pattern, stepfunctions can have task that is backed with a queue (like SQS). On the consumer side of the queue, the target resource (Ex: lambda, a containerized implementation, or an EC2 instance, processes the message and sends the status back to the state. This triggers the state machine to transition to the next step.

The aws-stepfunctions-sqs (a statemachine task that is backed by SQS) can be combined with the existing aws-sqs-lambda construct to complete this pattern (lambda specific version of the stepfunction callback pattern). Note: additional permissions are required to be assigned, in this case to the lambda function to be able to send back the task status to the statemachine.

As part of this construct, the proposal is to create a state machine fragment (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html) that would allow it to be chained to a state machine definition.

###Other https://zaccharles.medium.com/async-callbacks-with-aws-step-functions-and-task-tokens-9df97fe8973c https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html

###Readme here are more details about the actual construct with its README and architecture diagram.

• [x] :wave: I may be able to implement this feature request
• [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

[knihit]

Renaming it to aws-stepfunctionstasks-sqs, since it is the tasks which are backed by SQS not the stepfunctions itself. Also to make it composable with a larger statemachine definition, the construct should create a StateMachineFragment

[knihit]

@biffgaut, here are more details about the actual construct with its README and architecture diagram.