flyingmutant
Allow usage of rapid and synctest together
I am trying the new testing/synctest in Go 1.25. Unfortunately, it is not possible to use it together with rapid at the moment.
Trying to use synctest inside rapid:
package main
import (
"pgregory.net/rapid"
"testing"
"testing/synctest"
)
func TestSynctestInRapid(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
synctest.Test(t, func(t *testing.T) {
})
})
}
leads to this error:
./synctest_in_rapid_test.go:11:17: cannot use t (variable of type *rapid.T) as *testing.T value in argument to synctest.Test
Trying to use rapid inside synctest also fails though:
func TestSynctestInRapid(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
})
})
}
panic: testing: t.Deadline called inside synctest bubble [recovered, repanicked]
goroutine 38 [running, synctest bubble 1]:
testing.tRunner.func1.2({0x10284f9a0, 0x1028b3160})
/opt/homebrew/Cellar/go/1.25.2/libexec/src/testing/testing.go:1872 +0x190
testing.tRunner.func1()
/opt/homebrew/Cellar/go/1.25.2/libexec/src/testing/testing.go:1875 +0x2a8
panic({0x10284f9a0?, 0x1028b3160?})
/opt/homebrew/Cellar/go/1.25.2/libexec/src/runtime/panic.go:783 +0xf4
testing.(*T).Deadline(...)
/opt/homebrew/Cellar/go/1.25.2/libexec/src/testing/testing.go:2064
pgregory.net/rapid.checkDeadline({0x1028ba7b0?, 0x14000100c40?})
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/engine.go:98 +0xd0
pgregory.net/rapid.Check({0x1028ba6a0, 0x14000100c40}, 0x1028b1fa8)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/engine.go:120 +0x5c
babiel.invalid/rapid-synctest.TestSynctestInRapid.func1(0x14000100c40?)
/Users/Max/Documents/Sandbox/rapid-synctest/synctest_in_rapid_test.go:11 +0x30
testing.tRunner(0x14000100c40, 0x1028b1fa0)
/opt/homebrew/Cellar/go/1.25.2/libexec/src/testing/testing.go:1934 +0xc8
created by testing/synctest.testingSynctestTest in goroutine 37
/opt/homebrew/Cellar/go/1.25.2/libexec/src/testing/testing.go:2046 +0x1fc
FAIL babiel.invalid/rapid-synctest 0.225s
FAIL
Obviously the first option cannot work because rapid.T != testing.T. But the second option could be made to work if rapid did not call t.Deadline. However, AFAICT, there is no way for code to detect if it is inside a synctest bubble and should therefore not call t.Deadline. Catching the panic seems like a dirty hack.
I need to look at the code to see if this idea is the way to go, but from the cursory look it may be possible to have a rapid.SyncTest(rapid.T, func(rapid.T)) wrapper that calls synctest.Test inside with proper testing.T and converts it back into rapid one before invoking its closure argument. Calling Check inside synctest is definitely wrong.
it may be possible to have a rapid.SyncTest(rapid.T, func(rapid.T)) wrapper that calls synctest.Test inside with proper testing.T
Oh, I didn't realize that is possible. Maybe instead of making it synctest-specific, you could have a generic function to provide that testing.T, in case there are other testing libraries that people wanna use with rapid that require a testing.T? I guess that would be a lot more work since t.Run and such are not supported inside synctest, so you could leave those out.
Calling Check inside synctest is definitely wrong.
Yeah I thought it might be, since the whole timeout for minimization and such won't work at all
I have a prototype in 6be41f8ea26f6ea5695f3c7b83a94c36b8d71071, would be happy to have feedback on it.
I have a prototype in https://github.com/flyingmutant/rapid/commit/6be41f8ea26f6ea5695f3c7b83a94c36b8d71071, would be happy to have feedback on it.
@flyingmutant I gave it a try in my scenario and unfortunately I am getting a panic:
panic: rapid.invalidData("overrun") [recovered, repanicked]
goroutine 1756 [running (durable), synctest bubble 1]:
testing.tRunner.func1.2({0x1030df440, 0x10324e550})
/opt/homebrew/Cellar/go/1.25.3/libexec/src/testing/testing.go:1872 +0x190
testing.tRunner.func1()
/opt/homebrew/Cellar/go/1.25.3/libexec/src/testing/testing.go:1875 +0x2a8
panic({0x1030df440?, 0x10324e550?})
/opt/homebrew/Cellar/go/1.25.3/libexec/src/runtime/panic.go:783 +0xf4
pgregory.net/rapid.SyncTest.syncTestWithinRapid.func1.1()
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/synctest_enabled.go:72 +0x154
panic({0x1030df440?, 0x10324e550?})
/opt/homebrew/Cellar/go/1.25.3/libexec/src/runtime/panic.go:783 +0xf4
pgregory.net/rapid.(*bufBitStream).drawBits(0x1400025fac8?, 0x1400025fad0?)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/data.go:76 +0x144
pgregory.net/rapid.genFloat01(...)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/utils.go:27
pgregory.net/rapid.genGeom({0x10325b650?, 0x14000112b40?}, 0x4020000000000000?)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/utils.go:33 +0x5c
pgregory.net/rapid.genUintNBiased({0x10325b650, 0x14000112b40}, 0xe)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/utils.go:68 +0x9c
pgregory.net/rapid.genUintN({0x10325b650?, 0x14000112b40?}, 0x1400025fbc8?, 0x78?)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/utils.go:93 +0x28
pgregory.net/rapid.genIndex({0x10325b650?, 0x14000112b40?}, 0x1400025fc58?, 0x50?)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/utils.go:145 +0x30
pgregory.net/rapid.(*sampledGen[...]).value(...)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/combinators.go:172
pgregory.net/rapid.(*Generator[...]).value(0x0, 0x1400011e8f0?)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/generator.go:74 +0x50
pgregory.net/rapid.(*Generator[...]).Draw(0x103268420?, 0x1400011e8f0, {0x102f0c580, 0x6})
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/generator.go:47 +0x58
pgregory.net/rapid.(*stateMachine).executeAction(0x1400025fd90, 0x1400011e8f0)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/statemachine.go:127 +0x90
pgregory.net/rapid.(*T).Repeat(0x1400011e8f0, 0x14000451800)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/statemachine.go:61 +0x24c
gitlab.rz.babiel.com/provid/bavirt.checkRandomActions.func4(0x1400011e8f0)
/Users/Max/Documents/Repos/gitlab.rz.babiel.com/provid/bavirt/random_test.go:577 +0x38
pgregory.net/rapid.SyncTest.syncTestWithinRapid.func1(0x1400046b340)
/Users/Max/go/pkg/mod/pgregory.net/[email protected]/synctest_enabled.go:76 +0x184
testing.tRunner(0x1400046b340, 0x14000611950)
/opt/homebrew/Cellar/go/1.25.3/libexec/src/testing/testing.go:1934 +0xc8
created by testing/synctest.testingSynctestTest in goroutine 1755
/opt/homebrew/Cellar/go/1.25.3/libexec/src/testing/testing.go:2046 +0x1fc
This is a pretty involved state machine test. What I noticed is that the panic always happens after a t.Skip. I'll try to create a reproduction.
I could not reproduce this separately, and the panic only happens with a previous fail file. If I remove that file, no panic happens.
@mxey does the issue persist without using synctest and rapid.SyncTest?
@mxey does the issue persist without using synctest and rapid.SyncTest?
@flyingmutant I am not a 100% sure what you mean. The only changes between my test that worked and the one that panicked are
- updating rapid in go.mod to your https://github.com/flyingmutant/rapid/commit/6be41f8ea26f6ea5695f3c7b83a94c36b8d71071
- replacing my custom override that disables time.Sleep with the new rapid.SyncTest method
When I revert change 2, I do not get the panic.
For completeness, I also tried to only revert change 2 partially: removing my sleep override, but not using synctest. I also do not get a panic.