terraform-aws-control_tower_account_factory
terraform-aws-control_tower_account_factory copied to clipboard
aws-ia
Invoke Customisation Step Function exceeded maximum output limit
Terraform Version & Prov:
AFT Version:
(Can be found in the AFT Management Account in the SSM Parameter /aft/config/aft/version)
1.7.0
Terraform Version & Provider Versions
Please provide the outputs of terraform version and terraform providers from within your AFT environment
terraform version
1.3.7
terraform providers
provider[registry.terraform.io/hashicorp/aws] ~> 4.26
Bug Description
When I invoke a customisation step that involves many accounts, I run into a quota of Step Functions, namely the "Maximum input or output size for a task, state, or execution"
{
"Error": "States.DataLimitExceeded",
"Cause": "The state/task 'arn:aws:lambda:ap-southeast-1:123456789012:function:aft-customizations-identify-targets' returned a result with a size exceeding the maximum number of bytes service limit."
}
This is a combination of the large number of accounts (100+) and the amount of data we have in custom fields.
To Reproduce Steps to reproduce the behavior:
- Create large number of accounts.
- Fill account requests with sufficient amount of custom fields
- Invoke Customisation to target a large number of accounts
- See error
{
"Error": "States.DataLimitExceeded",
"Cause": "The state/task 'arn:aws:lambda:ap-southeast-1:123456789012:function:aft-customizations-identify-targets' returned a result with a size exceeding the maximum number of bytes service limit."
}
Expected behavior
To not fail
Related Logs
I can provide the lambda logs in a more private method if you need it.
Additional context
The recommended solution from the Step Functions documentation is to store the data somewhere else (e.g. S3) and pass around the ARN instead of raw JSON.
Hey @lawliet89!
Thank you for pointing out this issue! I went ahead and created an item in our backlog for this issue. Thanks!!
May I suggest you use a DynamoDB table rather than an S3 bucket for this? The former is lightweight and fast, the latter is more work to manage.
On Wed, 18 Jan 2023 at 20:45, hanafya @.***> wrote:
Hey @lawliet89 https://github.com/lawliet89!
Thank you for pointing out this issue! I went ahead and created an item in our backlog for this issue. Thanks!!
— Reply to this email directly, view it on GitHub https://github.com/aws-ia/terraform-aws-control_tower_account_factory/issues/298#issuecomment-1387675539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAA6OJBITDLLNCAVXHTFFTWTBB6DANCNFSM6AAAAAAT6RED3M . You are receiving this because you are subscribed to this thread.Message ID: <aws-ia/terraform-aws-control_tower_account_factory/issues/298/1387675539@ github.com>
Encountering the same issue, any delivery date for the fix ? Many thanks !
What's going on with this bug? This causes a hard fail on the entire step function and prevents any customization from being applied. This is causing a hard limit on the number of accounts we can manage with AFT even with limited custom field data.
This starts to cause hard failures in our environment with even a modest number of accounts processing at once... I've seen it fail with even 15 or so accounts. This is a major scalability problem in the platform and should be addressed.
We would be able to put in a PR for this issue.... if AFT was open for collaboration.
Just bumping this thread up as we are facing the same issue on account of having a large number of accounts
