Removing multiple newlines from EOF?

[acrewdson]

Hello,

When using terraform fmt I noticed that multiple newlines are not removed (ie collapsed down to a single newline) from ends of files. Would maintainers be open to a PR that would fix this?

Thanks!

[apparentlymart]

Hi @acrewdson,

For the moment hclwrite.Format (the basis of terraform fmt) is mainly focused on changes to horizontal whitespace: indenting, aligning, etc.

The way it's currently built is optimized for adjusting the whitespace between tokens rather than adding/removing tokens, and newlines are represented as tokens right now. However, truncating additional newline tokens from the end of the file as a subsequent step before returning seems like it'd be pretty non-disruptive, so seems reasonable to me to do it.

Another thing I've considered is a post-processing pass to copy the tokens into a new array while reducing any consecutive sequences of newline tokens down to a maximum of two. It does mean making another copy of the token list in order to be able to remove some tokens from the middle of it (unlike truncating the end, which is just a slicing operation) but for any reasonable HCL input file that copy would not be a significant burden, I think.

If hclwrite.Format is going to start making adjustments to vertical spacing as part of its scope, we should probably do both of these things at once, since I think they can be handled together as a single post-processing function called just before returning from hclwrite.Format.

[acrewdson]

Thanks @apparentlymart for this extremely helpful reply. I'll make an attempt at a PR that matches the goals you've outlined above and will request your feedback.

[apparentlymart]

I had a quick look at this myself today as part of some other work, and remembered why this is more complex than it first appears: the formatter is currently implemented as an in-place mutation of a list of tokens, with the goal of minimizing new allocations so that the formatter doesn't create a lot of memory pressure.

Unfortunately that means the current design doesn't currently allow the token sequence to shrink during formatting, which would be required in order to remove the newline tokens.

That doesn't mean we can't do this, but it does mean we'll need to do some internal refactoring to do it, and will want to try to design it so that as much as possible it is just truncating the existing list in-place, and ideally that it's very cheap to run this extra pass over a token sequence that is already formatted.

[acrewdson]

@apparentlymart I PR'd this change to hcl1 before realizing that Terraform uses hcl2 🤦‍♂ https://github.com/hashicorp/hcl/pull/351/files#diff-56a7a1546fe93b5da71f86b7ae456b12R64

I don't suppose you'd be open to adding something comparably ugly (but pragmatic?) over here https://github.com/hashicorp/hcl/blob/hcl2/hclwrite/public.go#L43

[acrewdson]

@apparentlymart I PR'd this change to hcl1 before realizing that Terraform uses hcl2 🤦‍♂ https://github.com/hashicorp/hcl/pull/351/files#diff-56a7a1546fe93b5da71f86b7ae456b12R64

I don't suppose you'd be open to adding something comparably ugly (but pragmatic?) over here https://github.com/hashicorp/hcl/blob/hcl2/hclwrite/public.go#L43

@radeksimko 👋 Any thoughts about the possibility of making the hacky TrimRight change to address this, at least until a prettier solution is implemented? It continues to be a persistent (if small) annoyance when using Terraform.