gotestsum
gotestsum copied to clipboard
Successful benchmark run is marked as failed
Hey all,
we observed gotestsum fail our benchmarks despite the benchmarks running fine. I tried to reproduce the issue and this was the smallest setup I could reproduce the issue with (using the latest gotestsum release v1.10.0):
Given a simple benchmark
import "testing"
func BenchmarkFuu(b *testing.B) {
l := 0
for i := 0; i < b.N; i++ {
l++
}
}
when run as e.g.
gotestsum --format standard-verbose --junitfile junit-results.xml --rerun-fails --rerun-fails-max-failures 10 --packages=. -- --bench=.
goos: darwin
goarch: arm64
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op
PASS
ok ... 0.571s
=== Failed
=== FAIL: . BenchmarkFuu (unknown)
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op
DONE 1 tests, 1 failure in 1.007s
it should not mark the test as failed. go version
is go version go1.20.4 darwin/arm64
.
I found https://github.com/gotestyourself/gotestsum/issues/62, is this related? Or do we use some incompatible configuration?