suite may race when its test creates goroutine

[lance6716]

in testify/suite/suite_test.go

// SuiteGoroutineRace is intended to test there should be no race when creating
// goroutine in tests.
type SuiteGoroutineRace struct{ Suite }

// TestSuiteRequireTwice...
func TestSuiteGoroutineRace(t *testing.T) {
	ok := testing.RunTests(
		allTestsFilter,
		[]testing.InternalTest{{
			Name: "TestSuiteGoroutineRace",
			F: func(t *testing.T) {
				suite := new(SuiteGoroutineRace)
				Run(t, suite)
			},
		}},
	)
	assert.Equal(t, true, ok)
}

func (s *SuiteGoroutineRace) TestGoroutine() {
	go func() {
		s.Equal(1, 1)
		s.Equal(1, 1)
		s.Equal(1, 1)
	}()
	go func() {
		s.Equal(1, 1)
		s.Equal(1, 1)
		s.Equal(1, 1)
	}()
	go func() {
		s.Equal(1, 1)
		s.Equal(1, 1)
		s.Equal(1, 1)
	}()
}

=== RUN   TestSuiteGoroutineRace
=== RUN   TestSuiteGoroutineRace
=== CONT  TestSuiteGoroutineRace
    testing.go:1152: race detected during execution of test
--- FAIL: TestSuiteGoroutineRace (0.01s)
=== RUN   TestSuiteGoroutineRace/TestGoroutine
==================
WARNING: DATA RACE
Write at 0x00c0000b8150 by goroutine 9:
  github.com/stretchr/testify/suite.(*Suite).SetT()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:36 +0xc4
  github.com/stretchr/testify/suite.(*SuiteGoroutineRace).SetT()
      <autogenerated>:1 +0x44
  github.com/stretchr/testify/suite.Run.func1.1()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:144 +0x2f4
  github.com/stretchr/testify/suite.Run.func1()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:159 +0x71f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /usr/local/go/src/testing/testing.go:1306 +0x47

Previous read at 0x00c0000b8150 by goroutine 10:
  github.com/stretchr/testify/suite.(*SuiteGoroutineRace).TestGoroutine.func1()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite_test.go:612 +0x36

Goroutine 9 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1306 +0x726
  github.com/stretchr/testify/suite.runTests()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:203 +0x18f
  github.com/stretchr/testify/suite.Run()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:176 +0x994
  github.com/stretchr/testify/suite.TestSuiteGoroutineRace.func1()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite_test.go:603 +0x44
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /usr/local/go/src/testing/testing.go:1306 +0x47

Goroutine 10 (finished) created at:
  github.com/stretchr/testify/suite.(*SuiteGoroutineRace).TestGoroutine()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite_test.go:611 +0x90
  runtime.call16()
      /usr/local/go/src/runtime/asm_amd64.s:625 +0x48
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:339 +0xd7
  github.com/stretchr/testify/suite.Run.func1()
      /home/lance/go/src/github.com/stretchr/testify/suite/suite.go:158 +0x71c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /usr/local/go/src/testing/testing.go:1306 +0x47
==================
    testing.go:1152: race detected during execution of test
    --- FAIL: TestSuiteGoroutineRace/TestGoroutine (0.00s)

--- FAIL: TestSuiteGoroutineRace (0.01s)


=== CONT  
    testing.go:1152: race detected during execution of test
    suite_test.go:607: 
        	Error Trace:	suite_test.go:607
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestSuiteGoroutineRace
    testing.go:1152: race detected during execution of test
=== CONT  
    testing.go:1152: race detected during execution of test
FAIL

[D3Hunter]

cannot reproduce locally

maybe paste complete code, some symbol is missing

[lance6716]

How about we add a Suite.Go(func()) that wraps a wait group? And we expect user to use this helper.

Our usage is in tests, we spawn a new goroutine to do some asynchronizing job, in the main test goroutine there is barrier to make sure the job is finished.

// in test function
go func() {
  err := someAsyncJob()
}()

but if we use suite to check the err, the checking may happen after the main test goroutine finished, so the cleaning up will cause race about checker.

[lance6716]

cannot reproduce locally

maybe paste complete code, some symbol is missing

the snippet should be put in testify/suite/suite_test.go

[lichunzhu]

How about we add a Suite.Go(func()) that wraps a wait group? And we expect user to use this helper.

SGTM.

To me, it's better to help users do the right thing(close all goroutines after a test) easier.

[dolmen]

Calling s.Equal after the test is finished (which is what usually happens if s.Equals is called from another goroutine) is unsupported.

More generally (beyond this case) parallel testing with suite is severely broken. Don't use concurrency with suite.