suite: Experimental - Fuzz test support

[asaha123]

Summary

Add support for fuzz tests as introduced in Go 1.18 (still beta) and documented here. Some additional implementation details are available in this blog post.

Changes

TBD

Motivation

Support fuzz tests in testify

Example usage

Example test:

package foo

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
	suite.Suite
	VariableThatShouldStartAtFive int
}

func (suite *ExampleTestSuite) SetupTest() {
	suite.VariableThatShouldStartAtFive = 5
}

func (suite *ExampleTestSuite) TestExample() {
	assert.Equal(suite.T(), 5, suite.VariableThatShouldStartAtFive)
	suite.Equal(5, suite.VariableThatShouldStartAtFive)
}

type ExampleFuzzSuite struct {
	suite.Suite
}

func (suite *ExampleFuzzSuite) FuzzAFunc() {
	f := suite.F()
	f.Add("foo")
	f.Fuzz(func(t *testing.T, aStr string) {
		// this fails the test run when run in fuzz mode
		//t.Fail()

		// this doesn't fail the test
		suite.EqualError(nil, "an error")
	})
}

func TestExampleTestSuite(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite))
}

func FuzzSuite(f *testing.F) {
	suite.Run(f, new(ExampleFuzzSuite))
}

With this modification to testify, the above test run in fuzz mode looks as follows:

$ go1.18beta1 test -fuzz=FuzzSuite -fuzztime=1s
fuzz: elapsed: 0s, gathering baseline coverage: 0/300 completed
fuzz: elapsed: 0s, gathering baseline coverage: 300/300 completed, now fuzzing with 8 workers
fuzz: elapsed: 1s, execs: 18028 (16323/sec), new interesting: 0 (total: 298)
PASS
ok  	test	1.308s

I expected the test to fail since i have suite.EqualError(nil, "an error').

In "unit test" mode, the test fails:

% go1.18beta1 test                             
--- FAIL: FuzzAFunc (0.00s)
    f_test.go:36: 
        	Error Trace:	f_test.go:36
        	            				value.go:556
        	            				value.go:339
        	            				fuzz.go:332
        	Error:      	An error is expected but got nil.
        	Test:       	FuzzAFunc
    f_test.go:36: 
        	Error Trace:	f_test.go:36
        	            				value.go:556
        	            				value.go:339
        	            				fuzz.go:332
        	Error:      	An error is expected but got nil.
        	Test:       	FuzzAFunc
--- FAIL: FuzzSuite (0.00s)
FAIL
exit status 1
FAIL	test	0.165s

[asaha123]

Thanks for the contribution! slightly_smiling_face

I've put down some questions to better understand the decisions you've made.

Thanks for the review so far. I have replied to the comments.

I suspect we may have run into an issue with how we handle assertions and report failures using testing.T value currently and it's workings in the context of fuzz tests: https://github.com/stretchr/testify/pull/1142/files#r821324429

[mcauto]

How's it going?

Go supports fuzzing in its standard toolchain beginning in Go 1.18. Native Go fuzz tests are supported by OSS-Fuzz.

https://go.dev/security/fuzz/