Resource FQN tokens aren't resolved in a user friendly way

[jsteinich]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

0.9.0. Any language.

Affected Resource(s)

Attempting to use .fqn on any resource when appending to it or using in an array (probably other cases as well). This mainly comes up when needing to use escape hatches.

Expected Behavior

Synthesized output is valid.

Actual Behavior

• Expected the start of an expression, but found an invalid expression token
• jsii.errors.JSIIError: Expected array type, got ""
• X is object with Y attributes

Important Factoids

Support for cross stack references caused a change in behavior. Previously .fqn actually return just a raw string (string token of string is the input string).

References

• https://discuss.hashicorp.com/t/python-escape-hatching-in-0-9-x/37202

[armandosorianopfs]

I just reached an scenario where .fqn is of type string, but it actually resolves into an array of objects, making it impossible to use it as value in any other object.

Just for the case, this is the scenario:

const webappService = new Service(this, 'k8s-service-webapp', {
  ...
  waitForLoadBalancer: true,
});

new DnsRecordSet(this, `webapp-dns-zone-record`, {
  ...
  rrdatas: [(webappService?.status.fqn as any)[0]?.load_balancer?.ingress?.ip], // won't be resolved properly
});

[DanielMSchmidt]

You would need to take terraform functions in this case. It should look sth like this: Fn.lookup(Fn.lookup(Fn.lookup(Fn.element(webappService.status.fqn, 0), "load_balancer"), "ingress"), "ip")

[armandosorianopfs]

@DanielMSchmidt you made my day!! thanks!!

If anyone reaches this issue with similar issue note that you need to use lookup to resolve object properties and element to resolve arrays. For my case, I've reached to this:

private getIpFromStatusFqn(fqn: string): string {
  const resolvedFqn = Fn.element(fqn as unknown as any[], 0);
  const loadBalancer = Fn.lookup(resolvedFqn, 'load_balancer', []);
  const loadBalancerFirstElement = Fn.element(loadBalancer, 0);
  const ingress = Fn.lookup(loadBalancerFirstElement, 'ingress', []);
  const ingressFirstElement = Fn.element(ingress, 0);

  return Fn.lookup(ingressFirstElement, 'ip', 'no-ip');
}

[paymog]

I'm running into a similar issue though using Fn.lookup and Fn.element isn't working well in all cases. Here's code that works well on version 0.8.6 (note the use of .fqn in the addOverride at the bottom)

    const dataAwsAvailabilityZonesAll =
      new aws.datasources.DataAwsAvailabilityZones(
        this,
        "allAvailableZones",
        {}
      );

    const efs = new aws.efs.EfsFileSystem(this, "efs-volume", {
      tags: {
        name: name,
        deployment: "ipfs",
      },
    });

    const efsMountTarget = new aws.efs.EfsMountTarget(
      this,
      "efs-mount-target",
      {
        dependsOn: [efs],
        count: 2,
        fileSystemId: efs.id,
        subnetId: "${aws_subnet.priv_subnet[count.index].id}",
        securityGroups: config.vpcSecurityGroups.ids,
      }
    );
    efsMountTarget.addOverride(
      "count",
      `\${length(${dataAwsAvailabilityZonesAll.fqn}.names)}`
    );

and here's my attempt at using Fn.element and Fn.lookup on version 0.12

    const zoneNames = Fn.lookup(dataAwsAvailabilityZonesAll.fqn, "names", undefined)
    efsMountTarget.addOverride(
      "count",
      Fn.lengthOf(zoneNames)
    );

but I still get issues like @armandosorianopfs gets:

[2022-08-02T09:47:39.574] [ERROR] default - ╷
│ Error: Invalid character
│
│   on cdk.tf.json line 271, in resource.aws_efs_mount_target.mercury-ipfs_efs-mount-target_B5A83491:
│  271:         "count": "${length(${data.aws_availability_zones.mercury-ipfs_allAvailableZones_45C25C25}.names)}",
│
│ This character is not used within the language.
goldsky-infra-dev  ╷
                   │ Error: Invalid character
                   │
                   │   on cdk.tf.json line 271, in resource.aws_efs_mount_target.mercury-ipfs_efs-mount-target_B5A83491 (mercury-ipfs/efs-mount-target):
                   │  271:         "count": "${length(${data.aws_availability_zones.mercury-ipfs_allAvailableZones_45C25C25 (mercury-ipfs/allAvailableZones)}.names)}",
                   │
                   │ This character is not used within the language.
                   ╵

⠇  Processing
[2022-08-02T09:47:39.592] [ERROR] default - ╷
│ Error: Invalid expression
│
│   on cdk.tf.json line 271, in resource.aws_efs_mount_target.mercury-ipfs_efs-mount-target_B5A83491:
│  271:         "count": "${length(${data.aws_availability_zones.mercury-ipfs_allAvailableZones_45C25C25}.names)}",
│
│ Expected the start of an expression, but found an invalid expression token.
goldsky-infra-dev  ╷
                   │ Error: Invalid expression
                   │
                   │   on cdk.tf.json line 271, in resource.aws_efs_mount_target.mercury-ipfs_efs-mount-target_B5A83491 (mercury-ipfs/efs-mount-target):
                   │  271:         "count": "${length(${data.aws_availability_zones.mercury-ipfs_allAvailableZones_45C25C25 (mercury-ipfs/allAvailableZones)}.names)}",
                   │
                   │ Expected the start of an expression, but found an invalid expression token.

[mutahhir]

@paymog I've been looking into this issue. I was hoping to get some more information, as I couldn't reproduce it exactly.

Why is the syntax error happening

From what I understand, the root of the problem is this pattern:

${length(${data.foo}.names)}

Our token system assumes that we're going to be used within a string and adds interpolation syntax ${} around the token. However, since you're already wrapping it within a ${}, that becomes a syntax error in Terraform.

The 0.12 example doesn't work

and here's my attempt at using Fn.element and Fn.lookup on version 0.12

   const zoneNames = Fn.lookup(dataAwsAvailabilityZonesAll.fqn, "names", undefined)
   efsMountTarget.addOverride(
     "count",
     Fn.lengthOf(zoneNames)
   );

This shouldn't emit the error you mentioned. I tried it out on my end, and it seems to be working fine on my end.

It outputs:

"count": "${length(lookup(data.aws_availability_zones.allAvailableZones, \"names\", []))}",

[paymog]

Thanks for looking into this @mutahir! I've long since resolved this issue and I don't quite remember how I did it.

Though, I suspect that the reason I was seeing that error on 0.12 was because 0.12 requires running tsc prior to diff/deploy while 0.8 didn't. I didn't realize that until quite a few hours into debugging the upgrade.

[jsteinich]

While more functionality has been added to reduce the need for using fqn, it still doesn't change that fqn is less useable to users in the cases that it would still be ideal to use it.

[mutahhir]

Ah, I didn't fully understand the core of the issue but went on from the last error message. My bad. Thanks for reopening @jsteinich, I'll take another look soon.

To clarify: The main issue is the inability to use .fqn within the interpolated section of a string. The token system's expectation that the resource fqn is being inserted into a non-interpolated terraform string is requiring users to jump through hoops to reference something that previously just required them to just do resource.fqn.

[M1kep]

Adding another example I'm currently running into while trying to setup a conditional expression, with variables in this case: If there's a better way I should be doing this please let me know.

Goal:

variable "a_var" {
    type = bool
}

variable "b_var" {}

output "test" {
    value = "${var.a_var ? var.b_var : null}"
}

Attempt:

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        a_var = TerraformVariable(self, "a_var")
        b_var = TerraformVariable(self, "b_var")

        TerraformOutput(self,
                        "test",
                        value="${" + f"{a_var.fqn} ? {b_var.fqn} : null" + "}"
        )

Generates:

...
  "output": {
    "test": {
      "value": "${${var.a_var} ? ${var.b_var} : null"
    }
  },
...

Workaround I'm currently using:

        TerraformOutput(self,
                        "test",
                        value="${" + f"var.{a_var.node.id} ? var.{b_var.node.id} : null" + "}"
        )