Unit testing failing to find resources for azurerm provider

[gbisaga]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

cdktf 0.8.6 terraform v1.1.3 on windows_amd64 nodejs v16.13.1 Note: I tested both directly under windows command prompt and under WSL, and I get the same results in both cases.

Affected Resource(s)

PublicIp is what I tested, but I expect all will be affected

Debug Output

N/A (problem in unit testing, it didn't generate a debug log file)

Expected Behavior

There are actually two problems here. I'm including them in the same issue because we think there's a good chance they have the same cause. The actual cdktf code deploys just fine; the problem is only in unit testing.

There are two kinds of test cases failing. We think they may be related.

    import { PublicIp } from "../.gen/providers/azurerm/public-ip";
    it ("is TEST CASE #1", () => {
      expect(
        Testing.synthScope((scope) => {
          new AppGatewayStack(scope, QA_PARAMS);   // AppGatewayStack does create a PublicIp
        })
      ).toHaveResource(PublicIp);
    })

    it ("is TEST CASE #2", () => {
      const app = new App();
      const stack = new AppGatewayStack(app, QA_PARAMS);
      expect(Testing.fullSynth(stack)).toMatchSnapshot();
    })

Expected behavior: Both tests should have passed.

Actual Behavior

TEST CASE #1 actual behavior: It fails with the error

 FAIL  __tests__/main-test.ts (12.786 s)
  My CDKTF Application
    Test the public IP
      × should have a public IP (59 ms)

  ● My CDKTF Application › Test the public IP › should have a public IP

    Type resource not found in stack

      58 |           // }
      59 |         })
    > 60 |       ).toHaveResource(PublicIp);
         |         ^
      61 |     })
      62 |   });
      63 |   // // All Unit testst test the synthesised terraform code, it does not create real-world resources

      at getAssertElementWithProperties (node_modules/cdktf/lib/testing/matchers.ts:108:13)
      at toHaveResourceWithProperties (node_modules/cdktf/lib/testing/matchers.ts:186:64)
      at Object.toHaveResource (node_modules/cdktf/lib/testing/adapters/jest.ts:72:65)
      at __EXTERNAL_MATCHER_TRAP__ (node_modules/expect/build/index.js:386:30)
      at Object.throwingMatcher [as toHaveResource] (node_modules/expect/build/index.js:387:15)
      at Object.<anonymous> (__tests__/main-test.ts:60:9)

TEST CASE #2 actual behavior: The first time you run it, it passes and generates a snapshot file (though, see below). The second time and after, it fails with the error below.

      ✕ should not change the snapshot (24 ms)

  ● AppGateway CDKTF Application › Test the public IP › should not change the snapshot

    expect(received).toMatchSnapshot()

    Snapshot name: `AppGateway CDKTF Application Test the public IP should not change the snapshot 1`

    Snapshot: "/tmp/cdktf.outdir.sMGHRL"
    Received: "/tmp/cdktf.outdir.fElhCW"

This second error actually makes sense in a way, because the snapshot file generated seems to be wrong. Rather than a representation of the stack resources, the __snapshots__/main-test.ts.snap file looks like this:

   // Jest Snapshot v1, https://goo.gl/fbAQLP

   exports[`AppGateway CDKTF Application Test the public IP should not change the snapshot 1`] = `"/tmp/cdktf.outdir.sMGHRL"`;

The JSON with actual stack resources IS being generated under that location.

Steps to Reproduce

Install dependencies and run npm run test

Important Factoids

References

• #0000

[DanielMSchmidt]

I tried to reproduce this, on OSX it seems to work, can you check if this production case fails as well for you? https://github.com/DanielMSchmidt/cdktf-issue-1506

[gbisaga]

Huh, it does work - I can run npm run test on that repo, and it works under windows. There's something weird going on here. One thing I did in your test is I changed the snapshot test to be a "normal" one:

    expect(
      Testing.synthScope((scope) => {
        new MyConstruct(scope, "myConstruct");
      })
    ).toMatchSnapshot();

Not only did this work, but the snapshot it generated looks correct. There's something weird going on with mine. I note that I include a number of other resources, as well as an Azure provider and S3 backend construct. I'll try to get it down to the minimum difference.

[gbisaga]

So I figured out what was wrong - sort of. At least, I figured out how to make it work, by comparing against your example. Problem was, when I was calling

        Testing.synthScope((scope) => {
          new AppGatewayStack(scope, QA_PARAMS);   // AppGatewayStack does create a PublicIp
        })

AppGatewayStack is defined as:

export class AppGatewayStack extends TerraformStack {
  constructor(scope: Construct, params: Parameters) { // , options: AppGatewayStackOptions
    super(scope, params.stackName);
    // etc.
  }
}

But you had it synthing a subclass of Construct, not TerraformStack. Now, given that TerraformStack extends Construct, I would expect that to work, also. But it doesn't. I instead modified it to:

        Testing.synthScope((scope) => {
          new AppGatewayConstruct(scope, QA_PARAMS);
        })
...
export class AppGatewayConstruct extends Construct {
  constructor(scope: Construct, params: Parameters) { // , options: AppGatewayStackOptions
    super(scope, `${params.stackName}-construct`);
    // etc.
   }
}

and now everything works!

[jsteinich]

Testing.synthScope internally creates a new stack. I don't believe there's been any testing done on nested stacks, so I'm not surprised that they don't work correctly. We could either make sure nested stacks work properly (probably best in the long term) or make Testing.synthScope more obvious that a stack can't be passed (probably fails with an App as well).

Your second test case comes down to Testing.fullSynth just always returns a temporary directory. Looks like the intent was to chain it with other functions (such as toBeValidTerraform. At a minimum there is some more documentation needed there. Should probably consider renaming the function as well.

[getflex-stephen]

Testing.synthScope internally creates a new stack. I don't believe there's been any testing done on nested stacks, so I'm not surprised that they don't work correctly. We could either make sure nested stacks work properly (probably best in the long term) or make Testing.synthScope more obvious that a stack can't be passed (probably fails with an App as well).

I don't think the immediate issue is best solved by making nested stacks work. Because the thing that's in the way here is the fact that Testing.synthScore creates a new stack regardless. I'm pretty sure the common case is going to be testing whether a Stack has specific resources, rather than a Construct.