gotestsum icon indicating copy to clipboard operation
gotestsum copied to clipboard

Successful benchmark run is marked as failed

Open noBlubb opened this issue 1 year ago • 2 comments

Hey all,

we observed gotestsum fail our benchmarks despite the benchmarks running fine. I tried to reproduce the issue and this was the smallest setup I could reproduce the issue with (using the latest gotestsum release v1.10.0):

Given a simple benchmark

import "testing"

func BenchmarkFuu(b *testing.B) {
	l := 0
	for i := 0; i < b.N; i++ {
		l++
	}
}

when run as e.g.

gotestsum --format standard-verbose --junitfile junit-results.xml --rerun-fails --rerun-fails-max-failures 10 --packages=. -- --bench=.      

goos: darwin
goarch: arm64
=== RUN   BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8   	1000000000	         0.3320 ns/op
PASS
ok  	...	0.571s

=== Failed
=== FAIL: . BenchmarkFuu (unknown)
=== RUN   BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8   	1000000000	         0.3320 ns/op

DONE 1 tests, 1 failure in 1.007s

it should not mark the test as failed. go version is go version go1.20.4 darwin/arm64.

I found https://github.com/gotestyourself/gotestsum/issues/62, is this related? Or do we use some incompatible configuration?

noBlubb avatar May 25 '23 09:05 noBlubb