Cross-platform issue as line-endings (CRLF <-> LF) not ignored

[jeroenhabets]

This ecs-cluster module does not ignore line endings in e.g. cluster.tpl so when I do terraform init plus terraform plan the "aws_launch_configuration" gets replaced because I am on a Mac (LF) and my colleague is on Windows (CRLF).

Rather than enforcing line endings using .gitattibutes (like we do in our repo for terraform), this provider should ignore line endings changes in e.g. cluster.tpl (user_data) (MUST HAVE) and referenced .pub keys (SHOULD HAVE). If hard to ignore line endings then always stripping CRLF->LF would provide a good solution.

module "ecs_cluster" {
  source = "infrablocks/ecs-cluster/aws"
  version = "4.2.0"

Reproduce

terraform init Convert to other line endings as during create (e.g. CRLF->LF) dos2unix .terraform/modules/ecs_cluster/user-data/cluster.tpl terraform plan => NOK: "aws_launch_configuration" gets replaced Convert to same line endings as during create (e.g. LF->CRLF) unix2dos .terraform/modules/ecs_cluster/user-data/cluster.tpl terraform plan => OK: no changes

NOTE: The same holds true for any referenced .pub key but we have a workaround through our /.gitattributes. It would be better if this module would ignore line endings here too (e.g. by always stripping CRLF->LF)

[tobyclemson]

Hi @jeroenhabets,

Thanks for raising this. However, I need a bit more information in order to work out the best way to resolve.

How are you fetching the module? Directly from Github? Or from the Terraform registry?

From what you've said, my understanding is that on fetch on a Windows machine, the line endings of some (all?) files are being converted to CRLF (from LF which they are in the repository) causing Terraform to detect drift on the next plan. Is this correct?

I thought git had a feature to manage automatic handling of line endings across platforms. Is it not possible for your Windows users to configure this via git configuration?

Thanks, Toby

[jeroenhabets]

Hi @tobyclemson,

Thanks for the swift response!

As shared in my report we fetch it from the Terraform registry (for Github the source would start with github.com AFAIK).

The issue is that Terraform fetches this module in the line-endings appropriate for the platform it's running on. For most files this doesn't matter. If cluster.tpl has different line-endings however this leads to a different user_data hash. AFAICT Terraform does not facilitate enforcing a line ending upon fetch (like Git does with .gitattributes).

The module would be insensitive if e.g. in the following line from asg.tf:

  cluster_user_data_template = coalesce(
    var.cluster_instance_user_data_template,
    file("${path.module}/user-data/cluster.tpl"))

the file would be forced to be LF using a replace:

  cluster_user_data_template = coalesce(
    var.cluster_instance_user_data_template,
    replace(file("${path.module}/user-data/cluster.tpl"), "\r\n", "\n"))

Note, that we'd have to warn users that created their ECS with Terraform on Windows they're clusters would be replaced the first run!

If that's unacceptable, introducing a bool variable, e.g. fix_crlf and only replace(... if true would make it backward compatible if the default is false.

The same thing could be done for key.tf and its public_key = file(var.cluster_instance_ssh_public_key_path) except the file is not part of the module so it's the users responsibility the file ending is correct.

I hope this makes it clear now.

[tobyclemson]

Thanks, that helps to clarify. Apologies, I missed the snippet showing how you were sourcing the module.

Could you also share your Terraform version?

Thanks, Toby

[tobyclemson]

According to Terraform's own issue log, Terraform does not change line endings in any case: https://github.com/hashicorp/terraform/issues/8467#issuecomment-512601057

As such, I'm confused what is causing this conversion and reluctant to modify the module to resolve this. I am happy to include a .gitattributes that would force those specific file to always have LF line endings, however, I'm not sure this will resolve your issue given that it is likely being triggered somewhere else in the toolchain.

Could there be anything else affecting the line endings?

[jeroenhabets]

Hi @tobyclemson ,

I'm using Terraform v1.3.5 on darwin_amd64.

A.o. because of Terraform's policy I stated "so it's the users responsibility the file ending is correct." for the public key file. Fine by me to mark that part WONTFIX.

However cluster.tpl is a file your (this) module provides. Adding a .gitattributes at the very least to force cluster.tpl line endings: cluster.tpl text eol=lf and that would fix it at least for people sourcing this via Github but that's the minority I'd presume.

TBH: I don't know enough about the Terraform registry to know for sure but from my observations it looks as if terraform just fetches modoules in the platforms line endings without a mechanism to steer this.

Just be aware, that if it's not fixed in the module than all multi-platform teams will be hit by this until it's too late (forcing them to maintain a copy of this module or switch to github if a .gitattributes were added).

[tobyclemson]

Hi @jeroenhabets,

Rather than forking the entire module, you should be able to control the template contents yourself by passing in cluster_instance_user_data_template to the module. You could copy the template contents directly and make whatever modifications in terms of line endings you need.

My hesitance to solve this in module is that if this is a problem caused by Terraform, it should be fixed by Terraform rather than requiring every module author to handle line endings themselves. If it is in fact a toolchain issue, it is the responsibility of module consumers to resolve the issue.

Thanks, Toby

[jonassvalin]

I will close this issue based on the final reply from @tobyclemson . If you feel like there's more to discuss then go ahead and re-open it @jeroenhabets .

Thanks, Jonas

[jeroenhabets]

@jonassvalin Thanks! I understand @tobyclemson 's concern and appreciate the workaround he offered.

Therefore I agree that for this module it's too late so keep this closed. Going forward it would be good to test a module on both Windows and Linux/macOS to detect and eliminate any changes due to line endings (e.g. replace CRLF->LF after reading a config file) as Terraform by design won't do that for you.