source introspection provider for terraform

[elasticdotventures]

Current Terraform Version

Terraform v1.2.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.75.2
+ provider registry.terraform.io/hashicorp/tls v4.0.0

Use-cases

source introspection means identifying which file, source, git-repo, etc. is currently being read/accessed "in scope" by the terraform cli, to determine where a resource definition originated.

the use-case:

aws "resource" "blabla" {
  // ...
   tags {
      definedIn="${introspection.source}"
   }
}

I use "${introspection.source}" as a placeholder, there would need to be some discussion about the specific syntax sugar.

The purpose of introspection is to include the source (such as a git repo, path, filename) where a definition can be found, this could conceivably even include many variables such as a line number, git tags, file hash - mostly for debugging/auditability of resources in the cloud. I am specifically trying to get the path and filename where a resource was defined to appear in the tags for auditing infrastructure & reporting.

Attempted Solutions

I do not believe this is possible presently (i've asked, but this doesn't seem possible without an enhancement to terraform)

Proposal

I would propose that such capability to be exceptionally useful be added in a manner similar to default_tags - where it can be specified at the provider level or equivalent mechanism. BUT still allowing terraform to do late stage interpolation when the resource is consumed from a .tf file. Other variables might include the module calling, or a trace of modules.

The default_tags approach means the tag could be easily and consistently propagated into all cloud resources that support tags.

This proposal may have applications outside of tags (such as logging).

Introspection variables might include diving into git and pulling not only checksums, tags, filenames, line numbers (for the resource).

References

• https://www.reddit.com/r/Terraform/comments/we2ojc/how_to_include_sourcetf_definition_of_resource_in/

[kmoe]

Thanks for the enhancement request. It is theoretically possible for providers to implement this functionality to a certain degree using ProviderMeta, but the data would still have to be passed in via the module. From Core's perspective we don't yet expose the introspection capability you describe.

[elasticdotventures]

From Core's perspective we don't yet expose the introspection capability you describe.

😁 okay, Thanks! @kmoe good to know. now I can stop searching for a solution/plugin somebody else built and start thinking about how I or somebody else might actually implement this.

I've discussed with peers and it seems like a useful potentially popular utility, no negative feedback yet.

There arose another approach to accomplish the same result which is better, cleaner & easier than the original proposal.

To enhance the terraform fmt command with --add-introspection-tags=tagName

The proposed parameter could do the interpolation as a lint step 'dsl formatter', targeting a specific tag (ex: "resource/tags"), outputting a pattern as specified for example --introspection-fmt="${resource.id}@${tffile.name}#${tffile.line_number}" which could be passed either via CLI or loaded from Environment.

🤔 The fmt directive is already making changes to the .tf syntax so it seems the least obtrusive place to put it in a workflow.

An example very boring file named main.tf with contents:

resource "aws_vpc" "example" {
}

run as such:

$ terraform fmt \
     --add-introspection-tags=tagName \
     --introspection-fmt="${resource.id}@${tffile.name}#${tffile.line_number}"

would output:

resource "aws_vpc" "example" {
    tags {
       tagName = "[email protected]#1"
    }
}

👆 I think that could be the first test case for a future feature. 😉

The obvious benefits of this approach are it this could be done & output to a separate branch as part of a ci/cd pipeline.
I'm also hoping the fmt --add-introspective-tags approach might be a bit easier (certainly will be to test), smaller crater size & more likely to get accepted as a pull request if I attempted to write it as a non-Go developer.

[Rumbles]

This has been discussed here, and while the poster has said he's only had good feedback so far, the only replies on his post about this idea say this is a bad idea. There is one reply suggesting another way to do this, which would probably work and wouldn't need changes to terraform.

The implementation suggested is also problematic as terraform fmt allows for no options/flags, as such we would all need to use this feature all the time if implemented as described, which would be a terrible idea.

I suspect the poster needs to find a better way to understand where resources live in their codebase

[elasticdotventures]

@Rumbles, James - Solo un punto de aclaración: mis compañeros 'peers' son personas con las que trabajo, no extraños de Internet al azar en Reddit. ¿Me consideras un compañero y un amigo? ¿Me trataste con respeto en Reddit? ¿Por qué te reconocería como un compañero?

I'm looking for a solution and asking for feedback.
If there is some easier way to trace back which object was created by which file it originated from is what I am asking, introspection is the equivalent of an IaC "source map" infrastructure traceability feature.

I'm not suggesting this become a default for fmt (*hence for a flag, in a future version). In an earlier draft of my post I originally suggested it could be a separate command - not fmt.

It was suggested to me privately (by a peer/co-worker) that it made more sense to run at the same time as fmt (outputting to a branch, prior to deployment) and so I lumped the idea with terraform fmt rather than propose a new top level command like "introspection" or something, the proposal didn't seem justified.

I acknowledge Reddit has been an immensely unhelpful place to ask for feedback on a feature idea. Good ideas are sometimes unpopular. I was not super articulate on reddit. I would like it to at least have it's discussion amongst the core team, would such a PR be useful to others -- maybe not to everybody, few if any features are useful to everybody.

Obviously reddit doesn't have the rules of conduct in place. But please let's keep the github discussion civil and on the merits or alternative ways to accomplish the same goal please.

[apparentlymart]

I think this is a similar sort of thing as what was discussed in #29024; just mentioning it here to create crosslinks between them for now, since I'm not sure if I'd consider it a duplicate or not (it seems to be approaching the problem from a slightly different perspective).