mu icon indicating copy to clipboard operation
mu copied to clipboard

Deploy on Acceptance stage failing

Open williamn opened this issue 5 years ago • 11 comments

I tried to follow the Quickstart tutorial but deployment on Acceptance stage is failing, here is the output from mu svc show:

Pipeline URL:	https://console.aws.amazon.com/codesuite/codepipeline/pipelines/mu-56-qs/view?region=ap-southeast-1
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+
|   STAGE    |  ACTION  |                 REVISION                 |                   STATUS                   |     LAST UPDATE     |
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+
| Source     | Source   | d7141868f0293916a8f1bb19422a192bdf52a7cc | Succeeded                                  | 2018-12-21 14:17:38 |
| Build      | Artifact |                                        - | Succeeded                                  | 2018-12-21 14:18:11 |
| Build      | Image    |                                        - | Succeeded                                  | 2018-12-21 14:19:16 |
| Acceptance | Deploy   |                                        - | Failed Build terminated with state: FAILED | 2018-12-21 14:26:01 |
| Acceptance | Test     |                                        - | -                                          |                   - |
| Production | Approve  |                                        - | -                                          |                   - |
| Production | Deploy   |                                        - | -                                          |                   - |
| Production | Test     |                                        - | -                                          |                   - |
+------------+----------+------------------------------------------+--------------------------------------------+---------------------+

Deployments:
+-------------+----------+------------------+---------------------+
| ENVIRONMENT | REVISION |      STATUS      |     LAST UPDATE     |
+-------------+----------+------------------+---------------------+
| production  | d714186  | CREATE_COMPLETE  | 2018-12-21 14:13:55 |
| acceptance  | d714186  | CREATE_COMPLETE  | 2018-12-21 14:13:55 |
+-------------+----------+------------------+---------------------+

Took a look at the CloudWatch logs, here is what I found:

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: ContainerInstances (AWS::AutoScaling::LaunchConfiguration) CREATE_FAILED API: autoscaling:CreateLaunchConfiguration You are not authorized to call EC2 Describe operations. It is required to perform CreateLaunchConfiguration operation.

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: Host2HostRuleEgress (AWS::EC2::SecurityGroupEgress) CREATE_FAILED Resource creation cancelled

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: ContainerInstances (AWS::AutoScaling::LaunchConfiguration) CREATE_FAILED API: autoscaling:CreateLaunchConfiguration You are not authorized to call EC2 Describe operations. It is required to perform CreateLaunchConfiguration operation.

[31mlogEventStatus ▶ ERROR [0m mu-56-environment-acceptance: Host2HostRuleEgress (AWS::EC2::SecurityGroupEgress) CREATE_FAILED Resource creation cancelled

I ran the command mu svc show using an AdministratorAccess granted IAM user. So I guess permission should not be the issue here.

williamn avatar Dec 21 '18 09:12 williamn

I'm also getting this on deploy.

Not sure if anything has changed here but it seems that autoscaling:CreateLaunchConfiguration needs some specific extra EC2 permissions to execute that don't seem to be present in the deploy-cluster policy.

waynerobinson avatar Jan 07 '19 02:01 waynerobinson

I also can't seem to work out a way to override that policy as the custom CloudFormation templates attempt to append the extra policies to the existing array and I can't just modify the CloudFormation template directly as the mu env upsert command will override the mu-iam-common template each time it's run.

waynerobinson avatar Jan 07 '19 02:01 waynerobinson

By the way, in case it's relevant, I'm using version 1.5.11-develop (because of some other errors that the current stable version didn't seem to be working with).

waynerobinson avatar Jan 07 '19 02:01 waynerobinson

OK, a couple of updates on this issue.

I have managed to test out some alternative permissions by changing the role manually after mu-iam-common was created (as CloudFormation doesn't automatically check for drift).

I needed to add both ec2:* and iam:CreateServiceLinkedRole to the deploy-cluster policy and it now seems to create correctly. I'm not sure what the exact extra ec2 permissions required are as I just went a bit nuclear on it. But it may just be the "describe" operations as listed in the error above.

waynerobinson avatar Jan 07 '19 03:01 waynerobinson

When changing ec2:* to ec2:Describe* the deploy step still works. So can probably limit these permissions in this case to just ec2:Describe* and iam:CreateServiceLinkedRole.

waynerobinson avatar Jan 07 '19 03:01 waynerobinson

Hi, I had the same issue but using mu 1.5.10. I've changed the ARN / Resource ID opt in for Fargate/ECS, as highlighted here: issues/414 and this appears to work without the above IAM changes.

chris-d-edwards avatar Jan 29 '19 17:01 chris-d-edwards

@chris-d-edwards I tried to opt in for the new ARN / Resource ID format but still getting the same error. I am using mu version 1.5.10

williamn avatar Feb 01 '19 11:02 williamn

Hi @williamn you need to opt out, so the boxes should be unchecked. Apologies my comment wasn't clear, if you read through the 414 issue, it states that the new format causes the issue. Did you do this for the root account (I had done this) or for your specific user which should be a the same IAM your using in the command line.

chris-d-edwards avatar Feb 01 '19 11:02 chris-d-edwards

I'm facing the same issue. Changing the ARN / Resource ID format had no impact (I was already opted out and opting in made no difference).

I've tried adding the permissions mentioned above to the role that was being assumed when the error is reported but haven't had any luck.

maxieduncan avatar Mar 31 '19 23:03 maxieduncan

Actually I was configuring the additional roles in the wrong place, once I added them to the mu-cloudformation-common-us-east-1 Role, the deployment worked as expected.

maxieduncan avatar Apr 01 '19 02:04 maxieduncan

Just tried the quickstart and had this same issue. Shouldn't the quickstart work out of the box?

drummerjoe avatar May 22 '19 02:05 drummerjoe