`cdktf get` keeps on `downloading and generating`, but never finishes

[alansikora]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

Terraform v1.0.11 on darwin_arm64 cdktf: 0.8.1

Affected Resource(s)

cdktf

Debug Output

GIST

Expected Behavior

Finish eventually

Actual Behavior

It keeps on running, never finishing the download (and it never populates the generated folder)

Steps to Reproduce

brew install terraform cdktf
mkdir folder
cd folder
cdktf init --template="go" --local

Add the aws provider to cdktf.json:

"terraformProviders": [
    "hashicorp/aws@~> 3.68.0"
],

cdktf get

Important Factoids

Trying to follow this tutorial

As simple as it gets

References

https://github.com/hashicorp/terraform-provider-aws/issues/22075

[alansikora]

I tried removing ~/.cdk, ~/.cdktf and ~/.terraform.d, without success. Can't seem to make it work, at all.

[alansikora]

So, I can see this error:

2021-12-06T19:28:29.495-0300 [ERROR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=arm64&os=darwin&signature=a42fdb3b-3a67-2616-77c0-17cadde8792f&version=1.0.11": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

But opening that URL directly works just fine:

And it appears to timeout in a fraction of a second.

[alansikora]

After being able to workaround the timeout issue, using a mobile network, it still won't work.

https://gist.github.com/alansikora/502a7a944a0906f9e915fae8d863cc91

The error is pretty much the same, which makes me very confused.

[alansikora]

This morning I tried again, and this happened:

https://gist.github.com/alansikora/7775381b5a5537860900dc07004b9dec

Error: jsii compilation failed with non-zero exit code: 255
  | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  | !!                                                                                  !!
  | !!  This software has not been tested with node v17.2.0.                            !!
  | !!  You may to encounter runtime issues, and should switch to a supported release.  !!
  | !!                                                                                  !!
  | !!  As of the current release, supported versions of node are:                      !!
  | !!  - ^12.7.0                                                                       !!
  | !!  - ^14.5.0                                                                       !!
  | !!  - ^16.3.0                                                                       !!
  | !!                                                                                  !!
  | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  | Error: RangeError: Maximum call stack size exceeded

The strange thing is, I'm not using node 17.2.0 (I don't even have it installed), I was using version 16.13.0. Anyway, I tried lowering to 16.3.0 and then the original error happened (never ending process).

EDIT:

This is a CDK thing, I fully removed NVM (this is not my main computer) and seems like CDK installs a not supported node version as a dependency 🤦

I will try to install everything without using homebrew and see how it goes.

EDIT 2:

No success. Still won't work at all, back to the original error.

[ansgarm]

Hi @alansikora,

does generating bindings for another provider work? For example for hashicorp/null@~> 3.1.0 instead of the AWS provider? Or does this produce the same error?

[robdennis]

I ran into this same issue using the python-pip template today. I did not try some of the workarounds referenced in the initial conversation, but I did try this:

does generating bindings for another provider work? For example for hashicorp/null@~> 3.1.0 instead of the AWS provider? Or does this produce the same error?

just now and with this cdktf.json value:

{
  "language": "python",
  "app": "python3 ./main.py",
  "projectId": "48f246e5-5d29-4ea6-9b8a-22b85c2252ff",
  "terraformProviders": [
    "hashicorp/null@~>3.1.0"
  ],
  "terraformModules": [],
  "codeMakerOutput": "imports",
  "context": {
    "excludeStackIdFromLogicalIds": "true",
"allowSepCharsInLogicalIds": "true"
  }
}

I was able to get this output in the console:

$ cdktf get
Generated python constructs in the output directory: imports

I was additionally able to get this result using hashicorp/google and with hashicorp/azurerm.

Just not with hashicorp/aws

EDIT: 2021-12-09

I was able to get AWS downloaded today (alongside google and azurerm)

[ansgarm]

Did cdktf get stop with any error for you? It is known that generating bindings for the AWS provider takes a substantial amount of time, especially for Golang.

How long did you wait for it to finish?

[robdennis]

On 2021-12-09 it took something like 5-10 minutes. I didn't get an error from cdktf get that time, just returned to console with Generated python constructs in the output directory: imports It's possible that the day I killed the process before it would have finished naturally.

For my use-case I moved over to the pre-built bindings available at https://pypi.org/project/cdktf-cdktf-provider-aws/ so I'm not longer using cdktf get as the way of bringing down dependencies

[lossanarch]

on 0.8.2 cdktf get is taking roughly 40-50m to generate golang bindings for me.

It regenerates existing files as well, so adding a provider is fairly tedious (i.e. it's no good prepopulating the generated dir, it will rebuild it all if I need to add a provider I haven't generated somewhere else before).

It's hard to see what it's doing; using CDKTF_LOG_LEVEL=debug results in about 4 extra output lines before it launches into the generation, but none during. The lack of feedback resulted in several failed attempts as I aborted at <10-20m the first few times.

[ansgarm]

Hi @lossanarch!

Regarding re-generation of existing bindings there is this open issue: https://github.com/hashicorp/terraform-cdk/issues/578, feel free to add a 👍 if you haven't already.

Provider generation relies on JSII that's why there are no logs that CDKTF_LOG_LEVEL=debug enables for the generation. There is this performance issue in JSII that affects all languages (in the TypeScript generation part) but Go seems to be particularly resource intensive (it also takes a lot of memory). That's something we should profile to check whether we can find similar issues there that we could report (and possibly fix) upstream in JSII.

[cikupin]

This problem also happens on me.

[alansikora]

@ansgarm some personal events kept me from answering, I'm sorry.

Using "hashicorp/null@~> 3.1.0" it works:

~/W/S/s/infrastructure ❯❯❯ cdktf get                                                                                                                     ✘ 1
Generated go constructs in the output directory: generated

The generated code depends on jsii-runtime-go. If you haven't yet installed it, you can run go mod tidy to automatically install it.
~/W/S/s/infrastructure ❯❯❯

[alansikora]

So, I waited over 1 hour, and it finished/worked.

This is a bit too much, I'd say haha

What can be done to improve this?

[ansgarm]

Hi @alansikora! Agreed, one hour is way to much. As noted earlier, we'd have to profile and improve the performance of JSII. Did you know about the pre-built provider packages we publish? Or is something blocking you from using these?

[mr-karan]

I waited for 2+ hours and eventually got OOM killed (16GB available) on my dev instance.

[arbfay]

My workaround has been to use this pre-built package. Add it to the dependencies in the Pipfile:

[packages]
cdktf-cdktf-provider-aws = "4.0.2"

This will also install matching cdktf and constructs packages, with the command pipenv install.

Then in the main.py script, just do a usual import as such:

from cdktf_cdktf_provider_aws import AwsProvider, ec2

Then no need to do cdktf get, just jump to cdktf plan or deploy.

Using pre-built packages is probably more pythonic anyways.

[DanielMSchmidt]

Prebuilt providers would help here, created https://github.com/hashicorp/terraform-cdk/issues/1482 to track this

[ansgarm]

Closing this as there have been different performance improvements to the code generation since, most recently through an upstream fix in JSII by @DanielMSchmidt.

https://github.com/hashicorp/terraform-cdk/issues/1886 contains a bit more context on this whole topic as well.

But as usual, if anything still feels off, don't hesitate to file a new issue and we'll look into it!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.