vscode-terraform meta: Support Terraform Test Framework

trafficstars

Problem Statement

Terraform 1.6 will ship with an updated and finalised terraform test command for running tests against your Terraform files. Tests will use a new file extension (.tftest.hcl & .tftest.json) and introduce new run, assert, ... blocks. Right now, a user will not get any syntax highlighting or advanced IDE features.

User Impact

The test command and test file syntax are aimed at module authors who wish to validate and test their shared modules. However, it is also possible to use the test command to validate root modules. So any user writing tests for Terraform code will benefit from this.

Expected User Experience

Syntax Highlighting

When you open a .tftest.hcl file, it should be highlighted in a similar way to how Terraform is highlighted today. This should happen automatically, without the user having to set the file type manually. The vscode-hcl extension will provide some basic syntax highlighting for the files, but we can enhance this with support in vscode-terraform.

Completion

A user can use auto-completion within a test file. Triggering completion at the top level will bring up all available blocks. Triggering completion within a block will bring up all available attributes and child blocks.

Hover

Hovering over any block, attribute or value will display additional information, similar to working within Terraform files.

References

Within a test file, a user can refer to resources and modules. The go-to definition should work for all resource and module sources and open the appropriate definition when triggered.

Completion should also bring up the existing resources which are defined in the Terraform files.

Semantic Tokens

We send accurate semantic tokens for all identifiers in a test file to further improve highlighting.

Formatting

Similar to how we format .tf files, we should implement formatting for .tftest.hcl files.

VS Code Integration

The Testing API allows VS Code extensions to discover tests in the workspace and publish results. Users can execute tests in the Test Explorer view, from decorations, and inside commands.

Proposal

Syntax Highlighting
- Create a new grammar for .tftest.hcl files
- Include the grammar and language support in VS Code
Schema
- Introduce a new schema for .tftest.hcl files
Language Server Support
- Handle & parse .tftest.hcl files
- Handle & parse .tftest.json files
- Enable autocompletion for .tftest.hcl files
- Enable hover for .tftest.hcl files
- Collect references for .tftest.hcl files
- Send semantic tokens for .tftest.hcl files
- Formatting for .tftest.hcl files
Terraform VS Code extension
- Integrate with the Testing API

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Jul 12 '23 12:07 dbanck

Related: https://github.com/hashicorp/vscode-terraform/issues/636

Maybe that issue can be broken down into the individual file formats but there may (or may not) be some higher-level work to do to support multiple formats easily/efficiently throughout all the libraries (hcl-lang + terraform-schema). Right now we support basically just *.tfvars and *.tf (JSON versions of those are mostly implied) and per the linked issue we may end up supporting many more, so we should ensure that this isn't just a matter of piling one edge case onto another. 😅

Jul 12 '23 15:07 radeksimko

note: as per https://developer.hashicorp.com/terraform/language/tests#syntax the extension is either .tftest.hcl or .tftest.json

Oct 09 '23 01:10 prototype7630

Now that the test framework was released with terraform 1.6, I think this issue is more pressing as people want to use it effectively in vscode!

Oct 20 '23 12:10 bew

Also don't forget the formatter. At the moment of writting this comment, the command terraform fmt work on tftest.hcl file. The format command of vscode should be able to support it also.

Oct 29 '23 16:10 martinOrigin8Cares

Question: I'm a newbie on the internal of terraform in general but if the terraform-cli already parce the "test framework" file, this code cannot be re-user for the language server? or even the "tokenization" that need to be perform for the command terraform fmt ?

Oct 29 '23 16:10 martinOrigin8Cares

@martinOrigin8Cares thanks for reminding me about formatting, I've added it to the list.

Under the hood, both Terraform and terraform-ls, use https://github.com/hashicorp/hcl to parse/tokenize the Terraform HCL files. After that step both tools work a bit differently. terraform-ls supports any Terraform version >= 0.12 and we load the specific schema/features for the user's version.

In terms of formatting: We currently use terraform fmt internally to format files, by passing the file contents via stdin (terraform fmt -), but we don't currently support .tftest.hcl files as a whole. So, when support for test files lands, formatting will be one of the first features we implement.

Oct 30 '23 14:10 dbanck

@dbanck Any progress on support for terraform test in the vscode extension?

I'm keen to help out if required - have dug into the code a bit, but not quite sure where to start.

Nov 29 '23 10:11 ghost

From what I can tell the following is required:

terraform-ls:

internal/terraform/ast/tests.go

Add AST for test files

// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0

package ast

import (
	"strings"

	"github.com/hashicorp/hcl/v2"
)

type TestsFilename string

func NewTestsFilename(name string) (TestsFilename, bool) {
	if IsTestsFilename(name) {
		return TestsFilename(name), true
	}
	return "", false
}

func IsTestsFilename(name string) bool {
	return strings.HasSuffix(name, ".tftest.hcl") ||
		strings.HasSuffix(name, ".tftest.json")
}

func (tf TestsFilename) String() string {
	return string(tf)
}

func (tf TestsFilename) IsJSON() bool {
	return strings.HasSuffix(string(tf), ".json")
}

type TestsFiles map[TestsFilename]*hcl.File

func TestsFilesFromMap(m map[string]*hcl.File) TestsFiles {
	mf := make(TestsFiles, len(m))
	for name, file := range m {
		mf[TestsFilename(name)] = file
	}
	return mf
}

func (tf TestsFiles) Copy() TestsFiles {
	m := make(TestsFiles, len(tf))
	for name, file := range tf {
		m[name] = file
	}
	return m
}

type TestsDiags map[TestsFilename]hcl.Diagnostics

func TestsDiagsFromMap(m map[string]hcl.Diagnostics) TestsDiags {
	mf := make(TestsDiags, len(m))
	for name, file := range m {
		mf[TestsFilename(name)] = file
	}
	return mf
}

func (td TestsDiags) Copy() TestsDiags {
	m := make(TestsDiags, len(td))
	for name, file := range td {
		m[name] = file
	}
	return m
}

func (td TestsDiags) AsMap() map[string]hcl.Diagnostics {
	m := make(map[string]hcl.Diagnostics, len(td))
	for name, diags := range td {
		m[string(name)] = diags
	}
	return m
}

func (td TestsDiags) Count() int {
	count := 0
	for _, diags := range td {
		count += len(diags)
	}
	return count
}

type SourceTestsDiags map[DiagnosticSource]TestsDiags

func (std SourceTestsDiags) Count() int {
	count := 0
	for _, diags := range std {
		count += diags.Count()
	}
	return count
}

internal/langserver/handlers/did_change_watched_files.go

// Line ~260
	if !ast.IsModuleFilename(fi.Name()) && !ast.IsVarsFilename(fi.Name()) && !ast.IsTestsFilename(fi.Name()) {
            ...
	}

vscode-terraform

src/extension.ts

activate > clientOptions > synchronize > fileEvents

       // Line ~82
        vscode.workspace.createFileSystemWatcher('**/*.tfvars'),
        vscode.workspace.createFileSystemWatcher('**/*.tftest.hcl'),
        vscode.workspace.createFileSystemWatcher('**/*.tftest.json'),

package.json

contributes > grammars

   // line ~113  
      {
        "language": "terraform-test",
        "scopeName": "source.hcl.tftest",
        "path": "./syntaxes/tftest.tmGrammar.json"
      }

syntaxes/tftest.tmGrammar.json

{
  "scopeName": "source.hcl.tftest",
  "name": "HashiCorp Terraform",
  "uuid": "d9db10d3-db70-48aa-8d44-f96ccbaa29f3",
  "fileTypes": [
    "tftest.hcl",
    "tftest.json"
  ],
  
  /// ???
  
}

Kind of stuck at the tftest.tmGrammar.json with the following:

Is the terraform.tmGrammar.json autogenerated, and can we generate a similar structure for the tftest.tmGrammar.json?
Due to the nested scope of some grammar in the tftest.hcl files, can we refer to the terraform.tmGrammar.json file or does the AST require that the embedded nodes are duplicated?
How do we effectively enable auto-complete for the references within assert scopes?
How do we effectively enable auto-complete for variables defined in both the test scope and the parent module scope?

Nov 29 '23 11:11 ghost

Hi @nagytech!

Thanks for the offer to help! I'm actively researching and working on the Terraform test integration and we plan to ship it early next year.

It will also require changes in terraform-schema and hcl-lang and some refactoring in terraform-ls. I will ticket out the work over the next month and we can see if anything comes up that you want to contribute.

For the static grammar, we will probably reuse the existing Terraform grammar and extend the list of blocks. This way we avoid maintaining and shipping another grammar file.

PS: If you want to keep digging, there are some poc branches in the repositories.

Dec 11 '23 11:12 dbanck

Hi!

Perhaps a bit out of topic.. I've being following a bit the testing subject whenever had some free time but there is something that bugs me and wanted to hear some opinions.

If we take an example some provider like aws which has underlaying tests implemented for each resource (and more). These tests verify that "all" implementations related to the resource and/or provider work as they should. In such case, what is the purpose of doing something like that (taken from https://developer.hashicorp.com/terraform/language/tests):

# main.tf

provider "aws" {
    region = "eu-central-1"
}

variable "bucket_prefix" {
  type = string
}

# This resource has all bucket*test.go related tests to make sure that it works.
resource "aws_s3_bucket" "bucket" {
  bucket = "${var.bucket_prefix}-bucket"
}

output "bucket_name" {
  value = aws_s3_bucket.bucket.bucket
}

Then why we need this?:

# valid_string_concat.tftest.hcl

variables {
  bucket_prefix = "test"
}

run "valid_string_concat" {

  command = plan

  assert {
    condition     = aws_s3_bucket.bucket.bucket == "test-bucket"
    error_message = "S3 bucket name did not match expected"
  }

}

(Same would be even if we didn't have the prefix variable.)

Dec 11 '23 11:12 ghs

Hi @ghs,

I think the Terraform core team in our Discuss is best suited to answer your question: https://discuss.hashicorp.com/c/terraform-core/27

Dec 11 '23 12:12 dbanck

@dbanck Hi Daniel, any updates on the status of this? Also happy to help if needed. Currently syntax validation is throwing errors for the new variables and run blocks, as well as other things included in the Terraform Test Framework added in v1.6.

At the minimum, is there a way to just disable the errors temporarily on my end? I'm trying to create an AWS workshop on using this new feature of Terraform (as part of an advanced pt.2 follow up to the Intro to Terraform on AWS Workshop, and currently I have to quit and re-open vscode every time I make any modification to the files. This makes it quite difficult to take any screenshots or make video explanations.

Jan 23 '24 19:01 novekm

@novekm You shouldn't get validation errors for Terraform test files today, because the extension doesn't (yet) claim the new extension .tftest.hcl. Only if you have manually changed the file association for test files to Terraform, you will get these errors. We don't recommend this because the LS doesn't know how to handle test files (yet).

You can get syntax highlighting for test files by installing the HashiCorp HCL extension.

Hope this helps! :)

Jan 24 '24 10:01 dbanck

You can add to the checklist the support for .tfmock.hcl files which have been added in 1.7 and the four new HCL blocks which go with that:

override_data
override_resource
mock_data
mock_resource

Also the content of those files is referenced in new mock_provider blocks in .tftest.hcl files.

Feb 01 '24 21:02 HenriBlacksmith

An interesting catch I hit in a project using terraform test.

We have a CI step to run terraform fmt -check -diff -recursive, which fails on unformatted .tftest.hcl files:

services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
--- old/services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
+++ new/services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
@@ -6,7 +6,7 @@
   command = plan
 
   assert {
-    condition = aws_ecs_service.this.desired_count > 0
+    condition     = aws_ecs_service.this.desired_count > 0
     error_message = "The desired_count must be greater than 0."
   }
 

Exited with code exit status 3

But unfortunately the Terraform extension isn't formatting these files yet so I'm manually having to run terraform fmt -recursive in the project, when I do happen to remember!

Feb 21 '24 16:02 sjparkinson

It has been almost a year and Terraform is currently at version 1.9. Any ETA on this issue?

Jul 23 '24 14:07 Taha-cmd

Still seems to be broken / missing as of 1.9.5

Aug 27 '24 11:08 sammcj

vscode-terraform vscode-terraform copied to clipboard

meta: Support Terraform Test Framework

Problem Statement

User Impact

Expected User Experience

Syntax Highlighting

Completion

Hover

References

Semantic Tokens

Formatting

VS Code Integration

Proposal

Community Note

terraform-ls:

internal/terraform/ast/tests.go

internal/langserver/handlers/did_change_watched_files.go

vscode-terraform

src/extension.ts

package.json

syntaxes/tftest.tmGrammar.json

vscode-terraform
vscode-terraform copied to clipboard