aws-deployment-framework icon indicating copy to clipboard operation
aws-deployment-framework copied to clipboard
verified publisher icon awslabs

Command did not exit successfully cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null exit status 1

Open skylagallm opened this issue 3 years ago • 1 comments

Hi,

I have an issue with my ADF version 3.0.4.

Starting from few days, I got an error while updating the deployment map, by the way no updates were made on a single component of ADF.

The output (and the error) is:

` 2022-10-05 19:59:45,491 | INFO | cdk_stacks.main | Pipeline creation/update of cur commenced | (main.py:34)

112 | Traceback (most recent call last): 113 | File "adf-build/cdk/generate_pipeline_stacks.py", line 37, in 114 | main() 115 | File "adf-build/cdk/generate_pipeline_stacks.py", line 33, in main 116 | PipelineStack(app, stack_input) 117 | File "/codebuild/output/src338995517/src/adf-build/jsii/_runtime.py", line 66, in call 118 | inst = super().call(*args, **kwargs) 119 | File "/codebuild/output/src338995517/src/adf-build/cdk/cdk_stacks/main.py", line 39, in init 120 | stack_input['input']["topic_arn"] = adf_notifications.Notifications(self, 'adf_notifications', stack_input['input']).topic_arn 121 | File "/codebuild/output/src338995517/src/adf-build/jsii/_runtime.py", line 66, in call 122 | inst = super().call(*args, **kwargs) 123 | File "/codebuild/output/src338995517/src/adf-build/cdk/cdk_constructs/adf_notifications.py", line 66, in init 124 | protocol=_sns.SubscriptionProtocol.EMAIL if '@' in _endpoint else _sns.SubscriptionProtocol.LAMBDA 125 | File "/codebuild/output/src338995517/src/adf-build/jsii/_runtime.py", line 66, in call 126 | inst = super().call(*args, **kwargs) 127 | File "/codebuild/output/src338995517/src/adf-build/aws_cdk/aws_sns/init.py", line 1419, in init 128 | jsii.create(Subscription, self, [scope, id, props]) 129 | File "/codebuild/output/src338995517/src/adf-build/jsii/_kernel/init.py", line 229, in create 130 | interfaces=[iface.jsii_type for iface in getattr(klass, "jsii_ifaces", [])], 131 | File "/codebuild/output/src338995517/src/adf-build/jsii/_kernel/providers/process.py", line 333, in create 132 | return self._process.send(request, CreateResponse) 133 | File "/codebuild/output/src338995517/src/adf-build/jsii/_kernel/providers/process.py", line 303, in send 134 | data = json.dumps(req_dict, default=jdefault).encode("utf8") 135 | File "/root/.pyenv/versions/3.7.6/lib/python3.7/json/init.py", line 238, in dumps 136 | **kw).encode(obj) 137 | File "/root/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 199, in encode 138 | chunks = self.iterencode(o, _one_shot=True) 139 | File "/root/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 257, in iterencode 140 | return _iterencode(o, 0) 141 | File "/codebuild/output/src338995517/src/adf-build/jsii/_kernel/providers/process.py", line 146, in jdefault 142 | raise TypeError("Don't know how to convert object to JSON: %r" % obj) 143 | TypeError: Don't know how to convert object to JSON: <SubscriptionProtocol.EMAIL: 'EMAIL'> 144 | Subprocess exited with error 1 145 |   146 | [Container] 2022/10/05 19:59:45 Command did not exit successfully cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null exit status 1 147 | [Container] 2022/10/05 19:59:45 Phase complete: BUILD State: FAILED 148 | [Container] 2022/10/05 19:59:45 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null. Reason: exit status 1 149 | [Container] 2022/10/05 19:59:45 Entering phase POST_BUILD 150 | [Container] 2022/10/05 19:59:45 Phase complete: POST_BUILD State: SUCCEEDED 151 | [Container] 2022/10/05 19:59:45 Phase context status code: Message: `

here my codebuild for deployment map:

version: 0.2 phases: install: runtime-versions: python: 3.7 nodejs: 10 pre_build: commands: - npm install [email protected] -g -y --quiet --no-progress - aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q -t ./adf-build build: commands: - cdk --version - chmod 755 adf-build/cdk/execute_pipeline_stacks.py adf-build/cdk/generate_pipeline_inputs.py adf-build/cdk/generate_pipeline_stacks.py - python adf-build/cdk/generate_pipeline_inputs.py - cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null - python adf-build/cdk/execute_pipeline_stacks.py

skylagallm avatar Oct 05 '22 20:10 skylagallm

Unfortunately I cannot replicate this error with the latest versions (v3.1.2 and the next upcoming one). What I am guessing is that a dependency used by that specific version of CDK changed.

What you could do is download the output of the CodeBuild container when it was still operational and compare that against its recent failed run. My guess is that a dependency changed and breaks this all of a sudden. When you find it, you need to add that to the adf-build/requirements.txt that is installed by the pip install -r adf-build/requirements.txt -q -t ./adf-build step. You can find this file in the aws-deployment-framework-bootstrap repo.

sbkok avatar Oct 13 '22 14:10 sbkok

I'm closing this issue as it has been inactive for a few weeks. This probably means that it is not reproducible or it has been fixed in the meanwhile.

Please reopen if you still encounter this issue with the latest stable version.

Thank you!

sbkok avatar Nov 04 '22 14:11 sbkok

@skylagallm - did you find a root cause for this problem? we are running into a very similar looking issue. Seems like maybe an unpinned dependency updated and is being pulled in?

atheiman avatar Dec 02 '22 19:12 atheiman

We were able to get past this by updating all dependencies to match what is currently in this ADF github repo:

  • Codebuild image aws/codebuild/standard:5.0
  • buildspec: npm install [email protected]
  • buildspec: nodejs: 14
  • buildspec: added the --no-version-reporting flag to the cdk synth --app adf-build/cdk/generate_pipeline_stacks.py command
  • updated adf-build/requirements.txt to match what is currently in this github repo
    • It might be a good idea to explicitly pin all python dependencies to avoid this in the future (pip freeze > requirements.txt)

atheiman avatar Dec 02 '22 20:12 atheiman

A customer reported running into this issue. In their case, they had modified quite a bit of ADF. Hence, they were not ready to upgrade to the latest release of ADF yet.

While searching for the root cause, it seems that the jsii package has very loose requirements on some of the dependencies they rely upon. The attrs and cattrs libraries would get installed with their latest releases for example.

Before I share how we resolved the issue, I would like to strongly recommend to upgrade to the latest version of ADF. v3.0.4 was released about three years ago. The libraries that ADF relies on have seen multiple security findings. Among others, see CVE-2022-40897, CVE-2021-33503, CVE-2020-26137, CVE-2022-42969, and CVE-2022-40898. On top of that numerous issues have been resolved since then.

If you insist, or need to resolve this temporarily before you apply the upgrade, this is how to fix it:

  1. In the management account.
  2. Go to the CodeCommit aws-deployment-framework-boostrap repository.
  3. Edit the following file: /adf-build/shared/requirements.txt.
  4. Append the following four dependencies to the end of the file:
attrs==19.3.0
cattrs==1.0.0
botocore==1.12.89
typing-extensions==3.7.4.3
  1. Commit this to the main branch (the master branch in v3.0.4).
  2. Wait for the bootstrap pipeline to succeed.
  3. Once that is finished, you can retry the pipeline generation and they should succeed then too.

Please reopen this ticket and ping me if this did not resolve your issue. Or open a new issue if you run into a slightly different problem after.

sbkok avatar Jan 31 '23 08:01 sbkok