Align output with `go test` to facilitate integration with editors

[ernesto-jimenez]

trafficstars

From PR #254:

Best thing to do to facilitate integration into tools like vim-go would be to align with the output from go test. Stdlib prints a single line per error as file.go:line: message.

As an example, vim-go has the following regex: let tokens = matchlist(line, '^\s*\(.\{-}\):\(\d\+\):\s*\(.*\)'). They have some logic to figure out the path of the file.

In order to align with go test and get our errors picked up by vim-go we need to output a single line of text matching the regex and showing a one-line summary.

e.g:

--- FAIL: TestExample (0.00s)
main_test.go:18: values should be equal

        Error Trace:    main_test.go:18
        Error:          Not equal: []string{"a", "b", "x"} (expected)
                                != []string{"a", "b", "y", "d"} (actual)

                        Diff:
                        --- Expected
                        +++ Actual
                        @@ -1,5 +1,6 @@
                        -([]string) (len=3 cap=3) {
                        +([]string) (len=4 cap=4) {
                          (string) (len=1) "a",
                          (string) (len=1) "b",
                        - (string) (len=1) "x"
                        + (string) (len=1) "y",
                        + (string) (len=1) "d"
                         }

However this would require reviewing all the error messages from testify and might require a breaking to include the one-liner error argument in Fail.

In any case, it's better to avoid adding the path. Plugins would already have logic to figure out the path based on the stdlib output. Adding the path can pollute the output considerably and make it less readable.

Furthermore, if we were to do it, I believe we should have it enabled by default rather than hiding it behind a flag or an env variable. That way it'll be integrated by the tool's default go test execution rather than require special configuration to have testify place nicely with editor plugins.

[ernesto-jimenez]

Steps required:

• [ ] Check whether adding the one-liner issues would require a breaking change.
• [ ] Define one-liner errors for all current assertions.

[jhillyerd]

+1

I like this idea. The sample output above looks clean, and might even allow editors to hide the more descriptive portion in a fold.

[benma]

What is the plan for this? I still jump to errors manually instead of automatically with the help of my editor ;)

Who should pick this up and prepare a PR?

Some remarks:

--- FAIL: TestExample (0.00s)
main_test.go:18: values should be equal

Note that the filename alone is not enough, it needs to at least include the path from the root of where go test has been called. The current testify output just has the filename, which does not tell me where the file actually is.

Define one-liner errors for all current assertions.

I think this is a nice-to-have, but in the short term we can get away with an empty or dummy short summary. The main goal of this for me is to be able to jump to the error location. I can already see the error description, so I don't rely on the short summary string. Future PRs could start adding nice short summaries per assertion error.

[andy-miracl]

Hi, Id love to see this taken forward as well. For, complete flexibility it would be nice if you could completely specify the format of the output.

[palsivertsen]

Anyone working on this?

Output for mock assertions also needs to be changed. Right now it's a panic with hard to parse filenames:

--- FAIL: TestMock (0.00s)
panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("Foo").Return(...) first, or remove the Foo() call.
	This method was unexpected:
		Foo()
		
	at: [mock_Fooer.go:13 foo.go:8 foo_test.go:20] [recovered]
	panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("Foo").Return(...) first, or remove the Foo() call.
	This method was unexpected:
		Foo()
		
	at: [mock_Fooer.go:13 foo.go:8 foo_test.go:20]

goroutine 7 [running]:
testing.tRunner.func1(0xc4201182d0)
	/usr/local/go/src/testing/testing.go:742 +0x29d
panic(0x6fc1c0, 0xc420048ef0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/stretchr/testify/mock.(*Mock).fail(0xc42007e410, 0x7930a8, 0xc1, 0xc420062a80, 0x4, 0x4)
	/home/pal/projects/go/src/github.com/stretchr/testify/mock/mock.go:235 +0x16d
github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc42007e410, 0x91be89, 0x3, 0x0, 0x0, 0x0, 0x3, 0x50, 0xc42007e410)
	/home/pal/projects/go/src/github.com/stretchr/testify/mock/mock.go:364 +0x978
github.com/stretchr/testify/mock.(*Mock).Called(0xc42007e410, 0x0, 0x0, 0x0, 0xc420035f60, 0x412188, 0x50)
	/home/pal/projects/go/src/github.com/stretchr/testify/mock/mock.go:333 +0x145
tmp.(*MockFooer).Foo(0xc42007e410)
	/home/pal/projects/go/src/tmp/mock_Fooer.go:13 +0x3e
tmp.DoFoo(0x7bdea0, 0xc42007e410)
	/home/pal/projects/go/src/tmp/foo.go:8 +0x31
tmp.TestMock(0xc4201182d0)
	/home/pal/projects/go/src/tmp/foo_test.go:20 +0x47
testing.tRunner(0xc4201182d0, 0x7971f8)
	/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x2e0
FAIL	tmp	0.006s

[jwatte]

This is still an issue btw. Why is this "input needed?"