hashicorp
Output still quoted after removing quote tokens?
Hi, I'm trying to generate HCL Data resources from arbitrary JSON while handling the transform myself .This is mostly going well, but when I attempt to use Terraform 0.12 variable syntax it doesn't work quite as anticipated.
Here's the input data: "provider": "${aws.development-west2}" As you can see, we're using the tf 0.11 interpolation syntax in the JSON to indicate that a variable is present.
Here's the transform function
func (p Passport) RemoveQuotesAndInterpolation(attributeName string, block *hclwrite.Block) *hclwrite.Block{
attr := block.Body().GetAttribute(attributeName)
toks := attr.BuildTokens(nil)
var tokNew hclwrite.Tokens
for _, t := range toks {
b := string(t.Bytes)
switch {
case strings.Contains(b, "$$"):
// TODO clean this up with regex
t.Bytes = []byte(strings.ReplaceAll(strings.ReplaceAll(strings.ReplaceAll(b, "$$", ""), "{", ""), "}", ""))
tokNew = append(tokNew, t)
case b == "\"":
continue
default:
tokNew = append(tokNew, t)
}
}
spew.Dump(tokNew)
attr.BuildTokens(tokNew)
return block
}
Here's the object used to compose the hclwrite.Block:
type AWSVPC struct {
Provider *string `mapstructure:"provider,omitempty" required:"false" cty:"provider" hcl:"provider"`
ID *string `mapstructure:"id" required:"false" cty:"id" hcl:"id"`
}
Here's the tokens that result from this function:
(hclwrite.Tokens) (len=4 cap=4) {
(*hclwrite.Token)(0xc00029f230)({
Type: (hclsyntax.TokenType) TokenIdent,
Bytes: ([]uint8) (len=8 cap=8) {
00000000 70 72 6f 76 69 64 65 72 |provider|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc00029f2c0)({
Type: (hclsyntax.TokenType) TokenEqual,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 3d |=|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc00029f140)({
Type: (hclsyntax.TokenType) TokenQuotedLit,
Bytes: ([]uint8) (len=21 cap=32) {
00000000 61 77 73 2e 64 65 76 65 6c 6f 70 6d 65 6e 74 2d |aws.development-|
00000010 77 65 73 74 32 |west2|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc00029f380)({
Type: (hclsyntax.TokenType) TokenNewline,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 0a |.|
},
SpacesBefore: (int) 0
})
}
Note lack of quote tokens. Yet when I review final output I receive this HCL:
data "aws_vpc" "west2" { provider = "aws.development-west2" id="blah" }
Rather than the expected output of
data "aws_vpc" "west2" { provider = aws.development-west2 id="blah" }
Here's the function used to assemble the slice of Data blocks for testing:
// AssembleHCL builds a slice of blocks into a string for use in testing
func AssembleHCLSlice(blocks []*hclwrite.Block) string {
dataFile := hclwrite.NewFile()
for _, b := range blocks {
dataFile.Body().AppendBlock(b)
}
return string(dataFile.Bytes())
}
And here's the token output when using an append for "\"" instead of a continue:
(hclwrite.Tokens) (len=6 cap=8) {
(*hclwrite.Token)(0xc000281230)({
Type: (hclsyntax.TokenType) TokenIdent,
Bytes: ([]uint8) (len=8 cap=8) {
00000000 70 72 6f 76 69 64 65 72 |provider|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc0002812c0)({
Type: (hclsyntax.TokenType) TokenEqual,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 3d |=|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc000281110)({
Type: (hclsyntax.TokenType) TokenOQuote,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 22 |"|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc000281140)({
Type: (hclsyntax.TokenType) TokenQuotedLit,
Bytes: ([]uint8) (len=21 cap=32) {
00000000 61 77 73 2e 64 65 76 65 6c 6f 70 6d 65 6e 74 2d |aws.development-|
00000010 77 65 73 74 32 |west2|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc000281170)({
Type: (hclsyntax.TokenType) TokenCQuote,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 22 |"|
},
SpacesBefore: (int) 0
}),
(*hclwrite.Token)(0xc000281380)({
Type: (hclsyntax.TokenType) TokenNewline,
Bytes: ([]uint8) (len=1 cap=1) {
00000000 0a |.|
},
SpacesBefore: (int) 0
})
}
What am I doing wrong that is resulting in quote tokens still being wrapped around my output rather than me receiving the unquoted output I expect after this token manipulation?
Thanks very much for taking a look!
Hi @geneshipt,
I'm afraid I'm not really following the chain of steps here to understand what you're trying to do and where it's going wrong.
However, one thing I did notice in isolation is that you have a statement attr.BuildTokens(tokNew) in the first function you shared, but it's weird to see a call to that function without assigning its result somewhere, because this function returns a new slice with the attribute's tokens appended to tokNew.
I think perhaps what you were intending to do here is to replace the attribute's tokens with a new set. If so, you could perhaps replace that statement with the following one instead:
block.Body().SetAttributeRaw(attributeName, tokNew)
However, SetAttributeRaw expects that second argument to be just the expression tokens for the attribute, rather than the whole attribute, so I think you'd also need to couple that with changing toks := attr.BuildTokens(nil) to toks := attr.Expr().BuildTokens(nil) so that the two are symmetrical, or else you'll end up with something confusing and invalid like provider = provider = aws.development-west2
I wrote some code that covered some similar ground as what you are doing here in a transient tool terraform-clean-syntax, so that might be a useful reference to adapt from. (That repository is archived because I later contributed most of it into the real terraform fmt, and so Terraform itself now implements rewrites like this on its own.)
