[BUG] [SM] SecurityPhase2(New MacieMemberAccount)? hang/failure for new accounts (Upgrade)

[AdilEzzaam-mcn]

Bug reports which fail to provide the required information will be closed without action.

Required Basic Info

• Accelerator Version: 1.5.0
• Install Type: Upgrade
• Upgrade from version: N/A
• Which State did the Main State Machine Fail in: Phase2

Describe the bug (A clear and concise description of what the bug is.) Client wants to do a state machine upgrade by including additional accounts to create within the config file. Failure is observed in SecurityPhase2 for alternate regions (UsEast_1 and ApSoutheast_2), while adding MacieMembers.

Failure Info

• What error messages have you identified, if any: State Machine, CodeBuild, CloudFormation PBMMAccel-Security-Phase2 precisely.
• What symptoms have you identified, if any: No pattern observed. Work around of re-running the state machine with apply all fixes the issue but the customer is not satisfied with the remediation.

Required files SEA Bug.zip Includes

• Config File
• CodeBuild failure US-EAST-1
• CodeBuild failure AP-SOUTHEAST_2
• CloudFormation US-EAST-1
• Custom SCPs

[Brian969]

• throttling issue which has started popping up on some new deployments, requires investigation.

[Brian969]

This is a new but reasonably frequent error, which is under investigation

• we've seen this occur when new AWS accounts were added to the top of the config file section rather than the bottom (docs indicate to add new items to the bottom, not the top)
• we've seen this occur when AWS accounts have been newly suspended
• we've seen this when the status of Macie in the security account is in a bad status (in any of the 17 regions) and needed to be manually cleaned up

Ultimately, we are investigating various root causes and mechanisms to avoid this from happening in all scenario's.

[Brian969]

• Last several releases reworked the back-off retry code, need to revalidate if this is still a challenge as it has not been raised recently.

[archikierstead]

Done