cdk-organizations
cdk-organizations copied to clipboard
pepperize
Organizations API not able to handle concurrent requests.
We are seeing an error when trying to use this library to attach Service Control Policies, and create Organizational Units. It seems like the AWS Organizations API is not able to handle concurrent requests.
Example code for SCPs:
export class appStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const org = new Organization(this, "master", {
featureSet: FeatureSet.ALL
})
org.attachPolicy(scps.getPolicy(this, scps.denyLeaveOrg))
org.attachPolicy(scps.getPolicy(this, scps.denyNewRegionsPolicy))
org.attachPolicy(scps.getPolicy(this, scps.denyOutsideEuCentral1AndUsEast1))
org.attachPolicy(scps.getPolicy(this, scps.denyCdkBootstrap))
}
}
CDK can synthesise templates and deploy a Cloudformation Stack. This fails with the following error:
Received response status [FAILED] from custom resource. Message returned: AWS Organizations can't complete your request because it conflicts with another attempt to modify the same entity. Try again later.
We also see the same error reported when trying to create multiple OUs that are at the same level.
Example code for OUs:
export class appStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const org = new Organization(this, "master", {
featureSet: FeatureSet.ALL
})
const security = new OrganizationalUnit(this, "security", {
organizationalUnitName: "Security",
parent: org.root,
})
const deployments = new OrganizationalUnit(this, "deployments", {
organizationalUnitName: "Deployments",
parent: org.root,
})
}
}
And again the same error of
Received response status [FAILED] from custom resource. Message returned: AWS Organizations can't complete your request because it conflicts with another attempt to modify the same entity. Try again later.
Would it be possible to add an exponential backoff to these requests?
Hey @kiwi-33 ,
yes, the AWS organization API is not able to create some resources concurrently.
Therefore you may use deployment.node.addDependency(security) to enforce the sequential creation of sibling resources. addDependency will then add CFN dependsOn to the stack.
See this example and construct dependencies
The CustomResources are using the SDK JavaScript V2, which should have exponential backoff configured by default.
We can also change the interfaces like organization.addOrganizationalUnit. Or using SNS FIFO provider instead of Lambda provider for the CustomResources.
WDYT
OK, thanks for the feedback. We will be able to work around this... but IMO I think that the consumers of the library should not have to worry about this API limitation.
If you are not going to change from the lambdas, it would be great to update the examples to show how to attach multiple SCPs at the same level, and explain how these dependencies are then defined.
I think a solution can be as follows:
Inside the Account class, you can look up other Account instances.
This can be looked up in the passed scope, which will assume all accounts are created in the same scope.
Or can lookup in the stack using Stack.of(this). This is technically less clean, as it is leaking thru layers. But probably a more robust approach.
This will assume all accounts are created in the same stack, which is also not exactly bulletproof, but probably equally as good as having an SNS, which will most likely be stack dependant as well.
const accounts = this.node.findAll().filter((node) => node instanceof Account)
Then take the last account and depend on that in the currently instantiated account.
A final option is to keep global state (array of Account instances, either as a global variable in the account.ts) or can be a static prop on the Account class.
Not sure about the ~~idempotency~~ stability of the findAll() call. Will it return the account instances in the order they were instantiated? If not, is there a way to get stable order?
I guess a simple solution is to just depend on every account that was created before current one. 🤷🏼 😁
Hey @moltar ,
thank you for your ideas. I've just created a poc with the Aspects: https://github.com/pepperize/cdk-organizations/pull/547
WDYT???
Hey, thanks for the suggestion. The example for Accounts is great. We are only using the library for managing OUs & SCPs and IMO, this needs to be extended for all OUs, SCPs, and SCPAttachments as well.
Interestingly, the first SCPAttachment should depend on the last SCP. We got OrganizationsAPI errors and we assumed it was because the first SCP was trying to be attached to an OU while there was also an API call to create a different SCP.
Once we defined that dependency, it worked fine.
(There is actually nothing in this Issue that references specifically "Account" - but generally the Organizations API's inability to handle concurrent requests.)
Hi,
Having the same issues with concurrency :(
Fairly new to CDK and would love to have an elegant solution similar to what I'm doing with terraform (Would have been great to pass an array of config objects to custom resource so it could take care of concurrency....).
Ideally I'd like to pass a config (buildConfig) for ous, accounts, scps etc with something like this:
{
"ProjectName": "man-org",
"AWSRegion": "ap-southeast-2",
"AppName": "man-org",
"Environment": "master",
"assume_role_name": "OrganizationAccountAccessRole",
"organization_accounts": {
"man-dev": {
"accountName": "man-dev",
"email": "[email protected]",
"roleName": "OrganizationAccountAccessRole",
"iamUserAccessToBilling": "IamUserAccessToBilling.ALLOW",
"parent": "dev",
"tags": {
"ou": "dev"
}
},
"man-network": {
"accountName": "man-network",
"email": "[email protected]",
"roleName": "OrganizationAccountAccessRole",
"iamUserAccessToBilling": "IamUserAccessToBilling.ALLOW",
"parent": "network",
"tags": {
"ou": "network"
}
},
"man-test": {
"accountName": "man-test",
"email": "[email protected]",
"roleName": "OrganizationAccountAccessRole",
"iamUserAccessToBilling": "IamUserAccessToBilling",
"parent": "test",
"tags": {
"ou": "test"
}
}
},
"organization_aws_service_access_principals": [
"aws-artifact-account-sync.amazonaws.com",
"ram.amazonaws.com",
"sso.amazonaws.com",
"access-analyzer.amazonaws.com",
"config.amazonaws.com",
"config-multiaccountsetup.amazonaws.com",
"securityhub.amazonaws.com"
],
"account_admin_delegation": {
"man-network": [
"sso.amazonaws.com"
],
"man-dev": [
"config.amazonaws.com",
"config-multiaccountsetup.amazonaws.com",
"access-analyzer.amazonaws.com"
]
},
"organization_enabled_policy_types": [
"SERVICE_CONTROL_POLICY"
],
"organization_feature_set": "ALL",
"ous_map_policies": {
"core": [
"deny-regions",
"leave-org",
"guardrails"
],
"dev": [
"deny-regions",
"leave-org",
"guardrails"
],
"lab": [
"deny-regions",
"leave-org",
"guardrails"
],
"network": [
"deny-regions",
"leave-org",
"guardrails"
],
"prod": [
"deny-regions",
"leave-org",
"guardrails"
],
"test": [
"deny-regions",
"leave-org",
"guardrails"
]
},
"tags": {
"repo": "https://github.com/man-Infrastructure/man-aws-org-main"
}
}
I figured a work-around for OUs and Accounts to cater for dependencies config:
// Create all OUs in the current organizations root
let ousArray: Array<OrganizationalUnit> = [];
let ousMap = new Map() // For future use?
for (let [key, value] of Object.entries(buildConfig.ous_map_policies)) {
let ou = new OrganizationalUnit(this, key, {
organizationalUnitName: key,
parent: organization.root,
importOnDuplicate: true,
});
ousArray.push(ou);
ousMap.set(key, ou)
// console.log("OUkey, OU 👉", key, ou);
}
// Organizations API is not able to handle concurrent requests
for(let i=0; i<ousArray.length; i++){
// console.log(ousArray[i]);
// Apparently: Received response status [FAILED] from custom resource. Message returned: AWS Organizations can't complete your request because it conflicts with another attempt to modify the same entity
if (i > 0 )
{
ousArray[i].node.addDependency(ousArray[i-1]);
}
}
But can't do the same workaround for
enablePolicyType/enableAwsServiceAccess/delegateAdministrator/attachPolicy and Policy?
as I'm getting - Property 'node' does not exist on type 'void' :(
Any idea how to do that? (Saw your PR - https://github.com/pepperize/cdk-organizations/pull/547 but can't figure how to make it work with the above enablePolicyType/enableAwsServiceAccess/delegateAdministrator/attachPolicy and Policy?
You may try out the aspect to chain the dependencies to sequentially deploy organization resources.
Update to 0.6.1 and add to your stack
import { DependencyChain } from "@pepperize/cdk-organizations";
Aspects.of(stack).add(new DependencyChain());
It should chain sequentially the mentioned resources. If you are fine with that solution, let's add it the Organization construct by default 😉
Thanks @pflorek !
I'll be testing all of those resources with DependencyChain next week (I'm away for a few days but will run the tests on Wed or Thursday and will let you know how it went) :)
I just tested DependencyChain to help with creating Account one at a time and it worked perfectly fine. Thank you everyone for the idea and implementation of this. 😊
Well some good news and some bad news on my side.
The enableAwsServiceAccess works just fine now:
organization.enableAwsServiceAccess("sso.amazonaws.com")
organization.enableAwsServiceAccess("aws-artifact-account-sync.amazonaws.com")
organization.enableAwsServiceAccess("ram.amazonaws.com")
organization.enableAwsServiceAccess("access-analyzer.amazonaws.com")
organization.enableAwsServiceAccess("config.amazonaws.com")
organization.enableAwsServiceAccess("config-multiaccountsetup.amazonaws.com")
organization.enableAwsServiceAccess("securityhub.amazonaws.com")
But when I try to use attachPolicy or delegateAdministrator I get 'Circular dependency' errors:
That's how it looks like for this piece of code:
for (let [key, value] of Object.entries(buildConfig.ous_map_policies)) {
let ou = new OrganizationalUnit(this, key, {
organizationalUnitName: key,
parent: organization.root,
importOnDuplicate: true,
});
// Add dependency to create Policies before OUs
ou.node.addDependency(leaveOrgPolicy)
// attachPolicy to ou
for(let i=0; i<value.length; i++){
if (value[i] === 'deny-regions') {
ou.attachPolicy(denyRegionsPolicy);
}
if (value[i] === 'guardrails') {
ou.attachPolicy(guardRailsPolicy);
}
if (value[i] === 'leave-org') {
ou.attachPolicy(leaveOrgPolicy);
}
}
// Tag OU
const mergedTags = deepmerge(buildConfig.tags, {"ou": key }, {
arrayMerge: (destination: any, source: any) => {
return [...destination, ...source];
}
});
tagResource(mergedTags, ou);
}
And if you wondered how ous_map_policies object looks like:
"ous_map_policies": {
"core": [
"deny-regions",
"leave-org",
"guardrails"
],
"dev": [
"deny-regions",
"leave-org",
"guardrails"
],
"lab": [
"deny-regions",
"leave-org",
"guardrails"
],
"network": [
"deny-regions",
"leave-org",
"guardrails"
],
"prod": [
"deny-regions",
"leave-org",
"guardrails"
],
"test": [
"deny-regions",
"leave-org",
"guardrails"
]
}
❌ man-org-master failed: Error [ValidationError]: Circular dependency between resources: [networkPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource28C7E69A, mantestCreateAccount67A39FEE, networkTagsTagResourceF60AB74A, corePolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicyEC527AEF, mantestTagsTagResourceB32E4798, corePolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy8D0E77E0, devPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource67EF2BF2, devPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicyE8664637, devPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource91874BC8, labPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceD0F6459D, testOrganizationProviderCBC7150C, devPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceD15CC595, mandevCreateAccountC963BBCC, corePolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource8CF359FD, corePolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource0A6C0FF9, prodPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy676C0276, networkPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource28746603, testTagsTagResource92162F2E, prodPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy0E41AEEB, corePolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy92C9815D, networkPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy4FBBAA8A, testPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource78815B63, prodPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceB4A3F7BB, testPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy1517DB73, testPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource5DB4600D, labTagsTagResource5E64019B, devTagsTagResourceD0F50801, devPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyD1FA17AA, prodOrganizationProviderDC6F8FD7, networkPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy31BA6DBE, mannetworkCreateAccount1BABA22E, testPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyEE8E0B2F, testPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource7FBD3028, mannetworkTagsTagResource1C043C63, mandevTagsTagResource62C57F89, labPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource99C28C93, prodTagsTagResourceBBB55045, labPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceC2CA2989, testPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy06F9130C, labPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyEC8FEDFF, labPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicyF926DA62, prodPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource7AD5514E, labPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy17FC8E5A, devOrganizationProviderFBEF1350, networkPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy03C9AB3B, networkPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource2AC95051, networkOrganizationProvider72BABE32, corePolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource398B1AD5, devPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy2CA4CDC0, prodPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCA55C08C, prodPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy8889F2A2, labOrganizationProvider15EF2273]
at Request.extractError (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'ValidationError',
time: 2022-09-15T07:29:55.291Z,
requestId: '869164a9-74a2-4afe-bc94-4c26b309c099',
statusCode: 400,
retryable: false,
retryDelay: 481.50551336050796
}
Circular dependency between resources: [networkPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource28C7E69A, mantestCreateAccount67A39FEE, networkTagsTagResourceF60AB74A, corePolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicyEC527AEF, mantestTagsTagResourceB32E4798, corePolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy8D0E77E0, devPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource67EF2BF2, devPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicyE8664637, devPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource91874BC8, labPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceD0F6459D, testOrganizationProviderCBC7150C, devPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceD15CC595, mandevCreateAccountC963BBCC, corePolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource8CF359FD, corePolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource0A6C0FF9, prodPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy676C0276, networkPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource28746603, testTagsTagResource92162F2E, prodPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy0E41AEEB, corePolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy92C9815D, networkPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy4FBBAA8A, testPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource78815B63, prodPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceB4A3F7BB, testPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy1517DB73, testPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource5DB4600D, labTagsTagResource5E64019B, devTagsTagResourceD0F50801, devPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyD1FA17AA, prodOrganizationProviderDC6F8FD7, networkPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy31BA6DBE, mannetworkCreateAccount1BABA22E, testPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyEE8E0B2F, testPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource7FBD3028, mannetworkTagsTagResource1C043C63, mandevTagsTagResource62C57F89, labPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResource99C28C93, prodTagsTagResourceBBB55045, labPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceC2CA2989, testPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy06F9130C, labPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicyEC8FEDFF, labPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicyF926DA62, prodPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource7AD5514E, labPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCustomResourcePolicy17FC8E5A, devOrganizationProviderFBEF1350, networkPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy03C9AB3B, networkPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResource2AC95051, networkOrganizationProvider72BABE32, corePolicyAttachmentmanorgmasterguardrailsDF13C312CustomResource398B1AD5, devPolicyAttachmentmanorgmasterleaveorgD8E1B6EBCustomResourceCustomResourcePolicy2CA4CDC0, prodPolicyAttachmentmanorgmasterguardrailsDF13C312CustomResourceCA55C08C, prodPolicyAttachmentmanorgmasterdenyregionsF5C8694DCustomResourceCustomResourcePolicy8889F2A2, labOrganizationProvider15EF2273]
Any idea how to fix it?
So as I've said with delegateAdministrator is the same issue:
for (let [key, value] of Object.entries(buildConfig.organization_accounts)) {
let account = new Account(this, key, {
accountName: value.accountName,
email: value.email,
importOnDuplicate: true,
iamUserAccessToBilling: IamUserAccessToBilling.ALLOW,
roleName: value.assume_role_name,
parent: ousMap.get(value.parent),
});
// Account Admin Delegation
for (let [key1, value1] of Object.entries(buildConfig.account_admin_delegation)) {
if (key1 === key) {
for(let i=0; i<value1.length; i++){
account.delegateAdministrator(value1[i]);
}
}
}
// Tag account
const mergedTags = deepmerge(buildConfig.tags, value.tags, {
arrayMerge: (destination: any, source: any) => {
return [...destination, ...source];
}
});
tagResource(mergedTags, account);
}
account_admin_delegation looks like this:
"account_admin_delegation": {
"man-network": [
"access-analyzer.amazonaws.com"
],
"man-dev": [
"config.amazonaws.com",
"config-multiaccountsetup.amazonaws.com"
]
}
and the error is:
❌ man-org-master failed: Error [ValidationError]: Circular dependency between resources: [mantestCreateAccount67A39FEE, mantestTagsTagResourceB32E4798, mandevDelegateConfigAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicyC4EF65CD, mandevDelegateConfigAmazonawsComDelegatedAdministratorCustomResource26784353, mannetworkDelegateAccessAnalyzerAmazonawsComDelegatedAdministratorCustomResource444FE373, mannetworkDelegateAccessAnalyzerAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicyC57B925F, mandevDelegateConfigMultiaccountsetupAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicy4A77FCFD, mannetworkCreateAccount1BABA22E, mannetworkTagsTagResource1C043C63, mandevDelegateConfigMultiaccountsetupAmazonawsComDelegatedAdministratorCustomResource5E64C1EE]
at Request.extractError (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'ValidationError',
time: 2022-09-16T04:17:03.439Z,
requestId: 'b95c714e-823c-47b3-b1f5-d540a4529733',
statusCode: 400,
retryable: false,
retryDelay: 144.69503779804006
}
Circular dependency between resources: [mantestCreateAccount67A39FEE, mantestTagsTagResourceB32E4798, mandevDelegateConfigAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicyC4EF65CD, mandevDelegateConfigAmazonawsComDelegatedAdministratorCustomResource26784353, mannetworkDelegateAccessAnalyzerAmazonawsComDelegatedAdministratorCustomResource444FE373, mannetworkDelegateAccessAnalyzerAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicyC57B925F, mandevDelegateConfigMultiaccountsetupAmazonawsComDelegatedAdministratorCustomResourceCustomResourcePolicy4A77FCFD, mannetworkCreateAccount1BABA22E, mannetworkTagsTagResource1C043C63, mandevDelegateConfigMultiaccountsetupAmazonawsComDelegatedAdministratorCustomResource5E64C1EE]
Tried a few things to resolve it but unsuccessfully :(
Hey @lkolchin ,
thanks for the error description. I could reproduce it easily as a test case. The reason for the circular dependency could be solved.
Previously while using attachPolicy or delegateAdministrator the PolicyAttachment and the DelegatedAdminstrator got the Account or the OrganizationalUnit as scope. If the sequential chaining aspect does policyAttachment.node.addDependency(account) it created a circular dependency to PolicyAttachment because PolicyAttachment is already part of the scope of Account. Either the chaining needs to be smarter or PolicyAttachment shouldn't be in the scope of Account.
Before:
- Stack
- Some Construct
- Account
- PolicyAttachment
- PolicyAttachment
- PolicyAttachment
- Account
- Some Construct
After:
- Stack
- Some Construct
- Account
- PolicyAttachment
- PolicyAttachment
- PolicyAttachment
- Some Construct
It should be fixed with 0.6.19
@pflorek
First of all - Thanks very much for promptly addressing those issues. I can confirm - it's fixed in 0.6.19
A couple of issues still need to be addressed (I think):
- When I tried to deploy 0.6.19 the first time it failed due to already enabled
SERVICE_CONTROL_POLICY. So, I had to disable it in the code, run the deployment (which would disable SERVICE_CONTROL_POLICY and fail) and then re-deploy again withorganization.enablePolicyType(PolicyType.SERVICE_CONTROL_POLICY)
man-org-master: creating CloudFormation changeset...
4:54:35 pm | CREATE_FAILED | Custom::Organizations_EnablePolicyType | OrganizationEnable...omResource79180BC7
Received response status [FAILED] from custom resource. Message returned: The specified policy type is already enabled. (RequestId: a84bdf90-887e-4f47-99aa-65f573c00c4e)
❌ man-org-master failed: Error: The stack named man-org-master failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: The specified policy type is already enabled. (RequestId: a84bdf90-887e-4f47-99aa-65f573c00c4e)
at prepareAndExecuteChangeSet (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:386:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CdkToolkit.deploy (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:219:24)
at initCommandLine (/Users/lkolchin/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/cli.ts:347:12)
The stack named man-org-master failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: The specified policy type is already enabled. (RequestId: a84bdf90-887e-4f47-99aa-65f573c00c4e)
- With the latest changes to role's self-assumption (https://aws.amazon.com/blogs/security/announcing-an-update-to-iam-role-trust-policy-behavior/)
AWS is warning about the need in Change to AWS IAM Role Trust Policy Evaluation
I think there is a case of - some Lambda functions call sts:AssumeRole with the target role being the very same role that the Lambda function has already been provided as part of its configuration. for
arn:aws:iam::xxxx:role/OrganizationAccountAccessRole
OrganizationAccountAccessRole - is configured in
let account = new Account(this, key, {
accountName: value.accountName,
email: value.email,
importOnDuplicate: true,
iamUserAccessToBilling: IamUserAccessToBilling.ALLOW,
roleName: value.assume_role_name,
parent: ousMap.get(value.parent),
});
Thank you :)
@lkolchin
The first is obvious. Due to change of the scope, a new resource is added, the old gets removed. We have to configure ignoreErrorCodesMatching that the resource won't fail. AwsSdkCall Need to figure out the correct error code of the organization API.
For the second issue I would you like to create a dedicated issue. It's currently still a warning?
@kiwi-33 , @moltar , @lkolchin please have a look at https://github.com/pepperize/cdk-organizations/pull/663 WDYT?
@kiwi-33 , @moltar , @lkolchin please have a look at https://github.com/pepperize/cdk-organizations/pull/663 WDYT?
@pflorek looks good to me . Thanks 😊