Testing: `toBeValidTerraform()` fails with "random" error messages

[DJAlPee]

Expected Behavior

I want to check, if my synthed stack is valid terraform syntax (and later, I also would like to plan the rollout) using toBeValidTerraform(). Tests should pass without any errors.

Actual Behavior

I get (randomly) one of the following error messages:

    Expected subject to be a valid terraform stack: Error: Command failed: terraform validate -json. Output: {
      "format_version": "1.0",
      "valid": false,
      "error_count": 1,
      "warning_count": 0,
      "diagnostics": [
        {
          "severity": "error",
          "summary": "registry.terraform.io/hashicorp/aws: the cached package for registry.terraform.io/hashicorp/aws 4.66.1 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file",
          "detail": ""
        }
      ]
    }
    .

    Expected subject to be a valid terraform stack: Error: Command failed: terraform validate -json. Output: {
      "format_version": "1.0",
      "valid": false,
      "error_count": 1,
      "warning_count": 0,
      "diagnostics": [
        {
          "severity": "error",
          "summary": "failed to read schema for aws_kms_alias.basic_kms_int_alias_C8FAFA52 in registry.terraform.io/hashicorp/aws: failed to instantiate provider \"registry.terraform.io/hashicorp/aws\" to obtain schema: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.",
          "detail": ""
        }
      ]
    }
    .

    Expected subject to be a valid terraform stack: Error: Command failed: terraform validate -json. Output: {
      "format_version": "1.0",
      "valid": false,
      "error_count": 2,
      "warning_count": 0,
      "diagnostics": [
        {
          "severity": "error",
          "summary": "Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.",
          "detail": ""
        },
        {
          "severity": "error",
          "summary": "Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.",
          "detail": ""
        }
      ]
    }
    .

Steps to Reproduce

stack1.spec.ts

import { Testing } from 'cdktf';
import { MyFirstStack, MyFirstStackProps } from './stack1';
import 'cdktf/lib/testing/adapters/jest';

describe('A stack, which uses all features', () => {
    const props = {
        // provide config for stack
    } as MyStackProps;

    // Other test e.g. toMatchSnapshot(), etc.

    it('check if the produced terraform configuration is valid', () => {
        const app = Testing.app();
        const stack = new MyFirstStack(app, 'my-app', props);
        // We need to do a full synth to validate the terraform configuration
        expect(Testing.fullSynth(stack)).toBeValidTerraform();
    });
});

stack2.spec.ts

import { Testing } from 'cdktf';
import { MySecondStack, MySecondStackProps } from './stack2';
import 'cdktf/lib/testing/adapters/jest';

describe('A stack, which uses all features', () => {
    const props = {
        // provide config for stack
    } as MyStackProps;

    // Other test e.g. toMatchSnapshot(), etc.

    it('check if the produced terraform configuration is valid', () => {
        const app = Testing.app();
        const stack = new MySecondStack(app, 'my-app', props);
        // We need to do a full synth to validate the terraform configuration
        expect(Testing.fullSynth(stack)).toBeValidTerraform();
    });
});

setup-jest.ts

const cdktf = require('cdktf');
cdktf.Testing.setupJest();

const matchers = require('jest-extended');
expect.extend(matchers);

jest config in package.json

{
    "jest": {
        "preset": "ts-jest",
        "clearMocks": true,
        "testEnvironment": "node",
        "moduleFileExtensions": [
            "js",
            "json",
            "ts"
        ],
        "rootDir": ".",
        "roots": [
            "assets",
            "src"
        ],
        "testRegex": ".*?(?=\\.spec).*?\\.ts",
        "transform": {
            ".+\\.(t|j)s$": "ts-jest"
        },
        "collectCoverage": true,
        "collectCoverageFrom": [
            "**/*.ts"
        ],
        "testPathIgnorePatterns": [
            "/node_modules/",
            ".*\\.ts\\.snap"
        ],
        "coverageDirectory": "../coverage",
        "coverageThreshold": {
            "global": {
                "functions": 95,
                "lines": 95
            }
        },
        "setupFilesAfterEnv": [
            "<rootDir>/setup-jest.ts"
        ]
    }
}

Versions

language: null cdktf-cli: 0.16.3 node: v16.18.1 terraform: 1.5.0 arch: arm64 os: darwin 22.5.0

Language is actually Typescript...

Providers

"@cdktf/provider-aws@^14.0.4": version "14.0.4" resolved "https://<self hosted Artifactory>/artifactory/api/npm/<NPM Cache>/@cdktf/provider-aws/-/provider-aws-14.0.4.tgz#be096f1e219c8c32a1f7ce801ec8f11f62e8c832" integrity sha512-/NGdzzVPvTchei/2O7AdId9WX3GequSQwlJ5wuAeMKCLlUBCqRPVU1ndsr48tkwvFdyefgK3Goh5/1xeiUOSdw==

"@cdktf/provider-random@^7.0.1": version "7.0.1" resolved "https://<self hosted Artifactory>/artifactory/api/npm/<NPM Cache>/@cdktf/provider-random/-/provider-random-7.0.1.tgz#04796d06b417322b2f5a62612c64b725b8c2505b" integrity sha512-GF14hSu7ZyE6j+FIAZcTp6kpBNWrY1A926VyPLXS1b0Zc9FczRwqQfNTivMX6qdnScDE+QzixuX+csOOoWLU2g==

Gist

No response

Possible Solutions

Because of that random behavior (sometimes even succeeds!), It feels like a race-condition issue...

Workarounds

Trying multiple times and praying...

Anything Else?

No response

References

No response

Help Wanted

• [ ] I'm interested in contributing a fix myself

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ansgarm]

Hi @DJAlPee 👋

How did you install Terraform? It seems like something is going on with some temporary or caching directory under the hood there and not directly CDKTF related.

Could you tell some more about your system and how you are executing the tests?

[DJAlPee]

Hi @ansgarm I have installed terraform binary on my Mac M1 (Pro) using homebrew:

✗ which terraform
/opt/homebrew/bin/terraform
✗ terraform --version
Terraform v1.5.0
on darwin_arm64
✗ cat ~/.terraformrc
plugin_cache_dir   = "${HOME}/.terraform.d/plugin-cache"
disable_checkpoint = true
✗ ls -la ~/.terraform.d
total 16
drwxr-xr-x   5 ap  staff   160 Jan  9 14:21 .
drwxr-x---+ 65 ap  staff  2080 Jun 26 10:30 ..
-rw-r--r--   1 ap  staff   326 Jan  9 14:21 checkpoint_cache
-rw-r--r--   1 ap  staff   394 Sep  6  2022 checkpoint_signature
drwxr-xr-x   3 ap  staff    96 Sep  6  2022 plugin-cache

What comes on top (but should not be an issue here?), that I don't have a cdktf.json file in my project, because synth of the terraform config is done within my project:

    const { SynthStack } = (await import('@cdktf/cli-core/src/lib/synth-stack.js'));
    const cdktfEntrypoint = path.join(cliDirname, 'infrastructure/main.js');
    const output = path.join(cwd, 'cdktf.out');
    const nodeCall = 'node ' + cdktfEntrypoint;
    const result = await SynthStack.synth(AbortSignal.timeout(100000), nodeCall, output);

[github-actions[bot]]

Hi there! 👋 We haven't heard from you in 30 days and would like to know if the problem has been resolved or if you still need help. If we don't hear from you before then, I'll auto-close this issue in 30 days.

[DJAlPee]

This is still valid. I have updated to latest version und still getting errors. There is now a new error popping up (sometimes):

    Expected subject to be a valid terraform stack: Error: Command failed: terraform validate -json. Output: {
      "format_version": "1.0",
      "valid": false,
      "error_count": 2,
      "warning_count": 0,
      "diagnostics": [
        {
          "severity": "error",
          "summary": "fork/exec .terraform/providers/registry.terraform.io/hashicorp/aws/5.10.0/darwin_arm64/terraform-provider-aws_v5.10.0_x5: exec format error",
          "detail": ""
        },
        {
          "severity": "error",
          "summary": "fork/exec .terraform/providers/registry.terraform.io/hashicorp/aws/5.10.0/darwin_arm64/terraform-provider-aws_v5.10.0_x5: exec format error",
          "detail": ""
        }
      ]
    }
    .

@ansgarm Would be great, if you could set this issue to a status, that won't close after 30 days without any new comments.

[DJAlPee]

@ansgarm and @xiehan it seems, that I have found the cause for this behavior!

It seems to be some kind of "race-condition" issue here. If there is only one test, which is calling toBeValidTerraform(), everything is fine. When there is a second stack, with a second test file, which uses toBeValidTerraform(), I get the errors, which I have already posted! Most properly because parallel execution causes some interferance?!

Is it possible to make toBeValidTerraform() (and most properly toPlanSuccessfully(), too) safe for parallel execution? Or do I have to fix this within my test setup? (But how?)

I have updated my issue reflecting this case a bit more detailed.

[DJAlPee]

--maxWorkers=1 seems to help, but this affects the performance of all tests. Whyever --maxConcurrency=1 won't help, even when the tests, which executes toBeValidTerraform(), are "marked" with it.concurrent.