tools: migrate test generation features to `gopls`

[emil14]

Is your feature request related to a problem? Please describe. I'm always frustrated when I click on "Generate test for function" and see "No tests were generated for ...".

Describe the solution you'd like I would like to see a reason of why there was no tests were generated. For example - "test for ... already exist".

Additional context It would be perfect to see a location of generated tests if there is already existing one. Or after successful generation.

[n1lesh]

@stamblerre I would like to work on this if no one's working.

[stamblerre]

Sounds good! /cc @hyangah to confirm that this would be a good thing to work on

[n1lesh]

Please correct me if I'm wrong, to generate tests vscode-go calls gotests lib internally as a child process.

However, gotests doesn't return any error if test(s) don't get generated other than "No tests were generated for...". The generateTest method (from gotests) ignores the func if test is already generated for it, and returns no error.

Probably modifying gotests to return an error for existing tests would be a better idea?

[stamblerre]

@n1lesh: Thanks for digging into the underlying cause! An alternative option may actually be to move the test generation logic into gopls itself, so that we can adjust the behavior more easily. Would you be willing to work on a gopls change that adds a command to generate tests instead? I know that's a bigger ask than this fix, but I'd be happy to offer pointers if you're interested.

[n1lesh]

@stamblerre I can work on the gopls change.

[h9jiang]

The Gopls will handle the add unit tests request following steps below:

• Gopls respond to code action request and add three new buttons including add unit test for selected functions, functions in this file and functions in this package. The implementation of those three buttons are another LSP rpc call add_unit_test to gopls.

• The add unit test for functions button will only show up when the select range overlap with a top level (file level not inline) function/method declaration.

• When user click on the button and request for unit test generation, the add_unit_test executeCommand rpc will be invoked and gopls will return WorkspaceEdit as response.

• As documented, WorkspaceEdit allows us to return TextDocumentEdit | CreateFile. Gopls will return CreateFile only if the test file does not exist already and will follow the right sequence (create file, then edit the text).

NOTE:

• The screenshot is only for demonstration purposes (is my local experiment).
• The test file name will follow the convention FILE.go -> FILE_test.go.
• The test name will follow the same convention as gotest. See method TestName defined in gotest repo for detail implementation.
• For un-exported function, the test function name is Test_function(t *testing.T).
• For exported function, the test function name is TestFunction(t *testing.T)
• For un-exported struct's method. Test_foo_function(t *testing.T) and Test_foo_Function(t *testing.T)
• For exported struct's method. TestFoo_function(t *testing.T) and TestFoo_Function(t *testing.T)
• The receiver's type (pointer or struct) does not make any difference to the test function name.

@hyangah @adonovan @findleyr

[findleyr]

Cool! A few notes:

Gopls respond to code action request and add three new buttons including add unit test

Let's just say "Add unit test for X". No need to prefix the code action with "Source:", since it is in the "Source Action" section.

When user click on the button and request for unit test generation, the add_unit_test grpc will be invoked and gopls will return WorkspaceEdit as response.

s/grpc/rpc. Also, the command does not return a workspace edit. Rather it calls the workspace/applyEdit server->client rpc.

Test function naming LGTM.

It may be simpler to start with just "Add unit test for function".

[adonovan]

Excited to use this! A couple of suggestions:

• The wording should be "Add" (since there may already be one), should avoid "unit", and should name the symbol. No "Source:" prefix is needed. So: Add test for function F or Add test for method T.M.
• It should only pop up if the symbol is exported. We shouldn't encourage writing tests of internals.

[h9jiang]

Add a source code action for single function. if the selected range is enclosed by a function decl or a method decl, we will provide this source code action to Add test for FUNCTION_NAME

Right now, the result of this action is returning a fake error.

[findleyr]

Cool! Should probably be "Add a test for Bar", but we can discuss that on the CL. This will be great.

[gopherbot]

Change https://go.dev/cl/621056 mentions this issue: gopls/internal/golang: add source code action for add test

[gopherbot]

Change https://go.dev/cl/621096 mentions this issue: gopls/internal/golang: create test file if not exist

[h9jiang]

Next CL: Following the existing implementation from gotest. When adding for a method, first declare a struct have all the fields as the receiver type.

For embedding type, we will choose the type name as fields name. E.g. Field Node for type ast.Node since struct bar does not provide a name for field which have type ast.Node.

func Test_foo_String(t *testing.T) {
	type fields struct {                  // <----- fields declaration.                
		Node       ast.Node 
		barinside  barinside
		inside     int
		insidecopy int
	}              
        ...
...

type bar struct {
	ast.Node
	barinside
	inside     int
	insidecopy int
}

type foo bar

func (*foo) String() string {return ""}

The end goal of the test looks below. (This is generate by the gotest)

func Test_foo_String(t *testing.T) {
	type fields struct {                          
		Node       ast.Node
		barinside  barinside
		inside     int
		insidecopy int
	}
	tests := []struct {
		name   string
		fields fields
		want   string
	}{
		// TODO: Add test cases.
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			f := &foo{
				Node:       tt.fields.Node,
				barinside:  tt.fields.barinside,
				inside:     tt.fields.inside,
				insidecopy: tt.fields.insidecopy,
			}
			if got := f.String(); got != tt.want {
				t.Errorf("foo.String() = %v, want %v", got, tt.want)
			}
		})
	}
}

[gopherbot]

Change https://go.dev/cl/621675 mentions this issue: gopls/internal/golang: add type decl including receiver's type fields

[h9jiang]

To update this issue with some decisions:

1. If the FILE_test.go does not exist, we will generate FILE_test.go with package PACKAGE_test. This follows the best practices where we want to test only the public APIs.
2. If the FILE_test.go does exist, we will generate test based on the specified package. If the package name is PACKAGE, we will allow generating un-exported functions or methods. If the package name is PACKAGE_test, we will generate only exported functions or methods. If the package name is not PACKAGE or PACKAGE_test, we will not generate any test for any functions/methods.
3. When adding test for a method, we will try to figure out the constructor instead of composite literals. For T's method, we will try to find any function that match one of the following signature (only allow if the constructor returns at most two values)

func NewT(....) T // returns T
func NewT(....) (T, any) // returns T, any 
func NewT(....) *T // returns *T
func NewT(....) (*T, any) // returns *T, any

4. If there are multiple return values from the constructor, we can try our best to handle if the second return value is error t, err := NewT(...) otherwise, drop using t, _ := NewT(...)
5. When adding test for PACKAGE, we will allow constructor newT.
6. After getting the return value from the functions/methods, we will leave the comparison to the user. The comparison is entirely determined by the context, we can never find an ideal comparison function between reflect.DeepEqual, ==, cmp.Diff. We will leave a comment and ask the user to write the comparison function. (except for the error return type, will explain in the next point)

      // TODO: update the condition below to compare with test.want.
      if true {
        t.Errorf("Foo(%d) = %s, want %s", test.in, got, test.want)
      }

7. If the function/method returns an error, there are some special handling. The test struct will have a field wantErr bool and correspondingly, a comparison block will be created to make sure the wantErr behavior matches with the returned error.

  got, got1, gotErr := Foo()
  if (err != nil) != test.wantErr {
    t.Errorf("Foo() got err %v, want error: %t", gotErr, test.wantErr)
  }

@hyangah @findleyr @adonovan

[adonovan]

All good, though in item 7 I suggest splitting the two cases as it leads to better error messages. (In Google readability reviews people sometimes say "but it's longer", but this is generated code so that's excuse won't fly. ;-)

  got, err := Foo(args)
  if err != nil {
        if !test.wantErr {
            t.Errorf("Foo(%v) failed: %v", args, err)
        }
        return
  }
  if test.wantErr {
      t.Fatalf("Foo(%v) succeeded unexpectedly", args)
  }
  ... compare got and test.want ...

[gopherbot]

Change https://go.dev/cl/621057 mentions this issue: gopls/internal/golang: generate test for function

[gopherbot]

Change https://go.dev/cl/622320 mentions this issue: gopls/internal/golang: create imports map honor x_test.go over x.go

[gopherbot]

Change https://go.dev/cl/623997 mentions this issue: gopls/internal/golang: add special handling for last return error

[gopherbot]

Change https://go.dev/cl/621816 mentions this issue: gopls/internal/golang: run testcases as subtests

[gopherbot]

Change https://go.dev/cl/620696 mentions this issue: gopls/internal/golang: support generating test for method w/o constructor

[gopherbot]

Change https://go.dev/cl/620697 mentions this issue: gopls/internal/golang: add missing imports in foo_test.go

[gopherbot]

Change https://go.dev/cl/626537 mentions this issue: gopls/internal/golang, go/ssa: remove unnamed input parameter

[gopherbot]

Change https://go.dev/cl/627355 mentions this issue: gopls/internal/golang: special handling for input context.Context

[gopherbot]

Change https://go.dev/cl/629756 mentions this issue: gopls/internal/golang: preserve copyright and build constraint

[gopherbot]

Change https://go.dev/cl/630119 mentions this issue: gopls/doc/features: add feature documentation for source.addTest

[gopherbot]

Change https://go.dev/cl/629978 mentions this issue: gopls/internal/golang: improve test package name selection for new file

[gopherbot]

Change https://go.dev/cl/632857 mentions this issue: gopls/internal/golang: Disable test generation for generic functions

[gopherbot]

Change https://go.dev/cl/633703 mentions this issue: gopls/internal/golang: Disable test generation for generic functions

[gopherbot]

Change https://go.dev/cl/692055 mentions this issue: gopls/internal/golang: show document after test generation