json2hcl icon indicating copy to clipboard operation
json2hcl copied to clipboard

Handling escaped characters when converting json to hcl

Open lorengordon opened this issue 7 years ago • 8 comments

When using functions where arguments are wrapped in double quotes, so there are both outer double quotes around the variable syntax and inner double quotes around the values, json2hcl handles the hcl to json part fine (other than the extra arrays already reported in a different issue), but when converting that json back to hcl the escape characters are not removed.

example.tf:

data "template_file" "iam_instance_policies" {
  count    = "${length(split(",", var.child["instance_policy_paths"]))}"
  template = "${file(join("/", list(path.module, replace(element(split(",", var.child["instance_policy_paths"]), count.index), "\n", ""))))}"

  vars {
    partition = "${data.aws_partition.current.partition}"
    account   = "${lookup(var.account_ids, terraform.workspace)}"
  }
}

cat example.tf | json2hcl -reverse:

{
  "data": [
    {
      "template_file": [
        {
          "iam_instance_policies": [
            {
              "count": "${length(split(\",\", var.child[\"instance_policy_paths\"]))}",
              "template": "${file(join(\"/\", list(path.module, replace(element(split(\",\", var.child[\"instance_policy_paths\"]), count.index), \"\\n\", \"\"))))}",
              "vars": [
                {
                  "account": "${lookup(var.account_ids, terraform.workspace)}",
                  "partition": "${data.aws_partition.current.partition}"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

cat example.tf | json2hcl -reverse | json2hcl:

"data" = {
  "template_file" = {
    "iam_instance_policies" = {
      "count" = "${length(split(\",\", var.child[\"instance_policy_paths\"]))}"

      "template" = "${file(join(\"/\", list(path.module, replace(element(split(\",\", var.child[\"instance_policy_paths\"]), count.index), \"\\n\", \"\"))))}"

      "vars" = {
        "account" = "${lookup(var.account_ids, terraform.workspace)}"

        "partition" = "${data.aws_partition.current.partition}"
      }
    }
  }
}

lorengordon avatar Sep 23 '17 16:09 lorengordon

Are these escape characters in the resulting HCL harmful and cause it to be invalid or is it just an aesthetic concern?

Acconut avatar Sep 28 '17 09:09 Acconut

The resulting HCL is invalid.

lorengordon avatar Sep 28 '17 09:09 lorengordon

I have debugged this issue a bit and, if it's a bug, it originates in Hashicorp's HCL library (https://github.com/hashicorp/hcl). Since I am not well versed in HCL on my own, could you explain to me when a double quote needs to be escaped in HCL? Only when it's not inside a ${ ... } or never?

Acconut avatar Oct 07 '17 21:10 Acconut

I'm on holiday this week and can't really test or research further as i don't have a computer. However, I cannot recall needing to escape double quotes at all in any of the HCL templates I've written.

lorengordon avatar Oct 08 '17 05:10 lorengordon

I've similar issue with my own automated tool that generates JSON configuration.

Here is the generated configuration:

{
  "resource":[
    {
      "bitbucket_hook":{
        "bamboo":{
          "events":[
            "repo:push",
            "pullrequest:fulfilled"
          ],
          "url":"${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}",
          "description":"Trigger Bamboo build: PRJ-KEY",
          "owner":"${var.owner}",
          "repository":"${bitbucket_repository.PRJ-KEY}"
        }
      }
    }
  ]
}

This configuration is converted to:

"resource" "bitbucket_hook" "bamboo" {
  "events" = ["repo:push", "pullrequest:fulfilled"]
  "url" = "${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}"
  "description" = "Trigger Bamboo build: PRJ-KEY"
  "owner" = "${var.owner}"
  "repository" = "${bitbucket_repository.PRJ-KEY}"
}

As you can see, the arguments for the format function are not de-escaped. It should be this way:

  "url" = "${format(var.bamboo_url, "PRJ-KEY", var.skip_branches ? "true" : "false")}"

beatcracker avatar Dec 26 '18 14:12 beatcracker

To be honest, I don't know HCL that well. Is "${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}" invalid?

Acconut avatar Dec 27 '18 22:12 Acconut

Sorry, should've made that clear: yes, it's not a valid HCL.

beatcracker avatar Dec 28 '18 00:12 beatcracker

This seems to be a problem in the upstream HCL package we use. So I reported the error there: https://github.com/hashicorp/hcl/issues/268. However, this has already been reported but there was no further activity: https://github.com/hashicorp/hcl/issues/233

Acconut avatar Dec 29 '18 19:12 Acconut