go
go copied to clipboard
golang
cmd/go: go test -cover & go test -coverprofile should always output a coverage
Description
Now in 1.10 when go test -cover supports multiple packages, I would expect it to print out a percentage for all packages (including those missing tests).
And for go test -coverprofile, I would expect all packages to be included in the calculated total.
Currently only packages that have at least one test (can be a *_test.go with only the package declaration) is included, see pkg2 below.
What version of Go are you using (go version)?
go version go1.10 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
Linux, amd64
What did you do?
go test ./... -cover
go test ./... -coverprofile cover.out; go tool cover -func cover.out
What did you expect to see?
? path/to/pkg1 0.001s coverage: 0.0% of statements [no test files]
ok path/to/pkg2 0.019s coverage: 0.0% of statements [no tests to run]
ok path/to/pkg3 0.371s coverage: 100.0% of statements
path/to/pkg1/pkg1.go:5: String 0.0%
path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 33.3%
What did you see instead?
? path/to/pkg1 [no test files]
ok path/to/pkg2 0.019s coverage: 0.0% of statements [no tests to run]
ok path/to/pkg3 0.371s coverage: 100.0% of statements
path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 50.0%
Change https://golang.org/cl/115095 mentions this issue: cmd/go/internal/test: always output a coverage
@kyroy noticed that this issue is very similar to #25492. It seems to me like both should be fixed at once, as they suggest changes in opposite directions. The other issue wants to change a 0.0% with a [no statements], and this one wants to change [no test files] with 0.0%.
/cc @bcmills @egonk
they suggest changes in opposite directions.
#25492 is about the case where there are no statements to cover (and thus the percentage is undefined). That's materially different from the case here, where pkg1 does have a statement and it is not covered: 0.0% coverage is well-defined and correct for a package with one statement and no tests.
My reading of this issue is that go test -cover should always output a percentage if it succeeded. The other issue is precisely about removing a 0.0% output in favor of something else.
My reading of this issue is that
go test -covershould always output a percentage if it succeeded.
That's the current issue title, but that behavior seems clearly wrong if there are no statements to cover: 0/0 is not a well-defined percentage.
(I guess we could output NaN%, but that seems strictly less helpful than [no statements].)
I like the idea of a consistent output that would be created by just implementing this issue.
Another approach is, as in #25492, for all statements it is true that they are covered by the tests. So one could argue for 100% which is obviously mathematically incorrect.
I am also the creator of the CL. Happy to implement a solution that results from the discussion :)
Change https://golang.org/cl/122518 mentions this issue: cmd/go: revert "output coverage report even if there are no test files"
FYI, f7248f05946c1804b5519d0b3eb0db054dc9c5d6 brought back the pkg{1,2,3} folders inside cmd/go/testdata/testcover that was originally reverted by https://github.com/golang/go/commit/7254cfc37b3a93a6e83dae22c4bfd6f777edb97e. Those files are unused as of now.
One can use -coverpkg=./... to get accurate results:
$ tree .
.
├── coverage.out
├── package0.go
├── package0_test.go
├── package1
│ ├── package1.go
│ └── package1_test.go
└── package2
└── package2.go
2 directories, 6 files
Without -coverpkg=./...:
$ go test -v -coverprofile=coverage.out ./...
=== RUN TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 100.0% of statements
ok github.com/rabadin/brokencoverage 0.007s coverage: 100.0% of statements
=== RUN TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 100.0% of statements
ok github.com/rabadin/brokencoverage/package1 0.006s coverage: 100.0% of statements
? github.com/rabadin/brokencoverage/package2 [no test files]
$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3: min 100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3: add 100.0%
total: (statements) 100.0%
With -coverpkg=./...:
$ rm coverage.out
$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok github.com/rabadin/brokencoverage 0.006s coverage: 33.3% of statements in ./...
=== RUN TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok github.com/rabadin/brokencoverage/package1 0.006s coverage: 33.3% of statements in ./...
? github.com/rabadin/brokencoverage/package2 [no test files]
$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3: min 100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3: add 100.0%
github.com/rabadin/brokencoverage/package2/package2.go:3: mult 0.0%
total: (statements) 66.7%
Using -coverpkg does not actually help if the package is never called in any test at all. -coverpkg is a cover-up that allows packages from a different package to provide coverage for this package.
So, say package a has no tests, and thus no coverage. But package b has tests, and those tests call into a. As a result, if you start using a -coverpkg that covers both packages, the tests from package b will report its coverage of package a as coverage for a.
However, if we have a package c that is never called by either a or b then that package will still be left out in the dry, even if it would match the -coverpkg given.
project$ find . -name "*_test.go"
project$ go test -coverpkg=./... -coverprofile=coverage.out ./...
? project [no test files]
? project/sub/dir1 [no test files]
? project/sub/dir2 [no test files]
project$ cat coverage.out
mode: set
project$
Additionally, it is misleading to say that because b has called into a that that coverage of a counts as tests on a, because they are not unit tests of a’s functionality, and b is fundamentally not responsible for testing of a.
@puellanivis:
Using -coverpkg does not actually help if the package is never called in any test at all.
I'm not sure this is true. Here is the code from the example above:
find . -name "*.go" -exec echo '******** {}' \; -exec cat {} \;
******** ./package2/package2.go
package package2
func mult(a int, b int) int {
return a * b
}
******** ./package0_test.go
package brokencoverage
import "testing"
func TestMin(t *testing.T) {
if min(3,5) != 3-5 {
t.Errorf("min is broken")
}
}
******** ./package1/package1_test.go
package package1
import "testing"
func TestAdd(t *testing.T) {
if add(3,5) != 3+5 {
t.Errorf("add is broken")
}
}
******** ./package1/package1.go
package package1
func add(a int, b int) int {
return a + b
}
******** ./package0.go
package brokencoverage
func min(a int, b int) int {
return a - b
}
as you can see nothing is calling package2/package2.go:mult.
I don’t know exactly what is different, but even with -coverpkg=./... my project with no tests still reports no packages:
project-with-no-tests$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
? project-with-no-tests [no test files]
? project-with-no-tests/subpackage [no test files]
project-with-no-tests$ go tool cover -func=coverage.out
total: (statements) 0.0%
While adding just a single _test.go with just the package name yields coverage details for a package.
Meanwhile, if I add a reference to package2.Mult in package0_test.go suddenly, with -coverpkg=./... we report that there is coverage of package2.Mult, even though it is not actually covered by any unit tests. (It would make sense to include such coverage in integration tests though, as there you are testing the system as a whole, but unit tests should never count coverage provided by any other package.)
broken-coverage$ cat package0_test.go
package brokencoverage
import (
"testing"
"github.com/puellanivis/broken-coverage/package2"
)
func TestMin(t *testing.T) {
if min(3,5) != package2.Mult(-1, 2) {
t.Errorf("min is broken")
}
}
broken-coverage$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 66.7% of statements in ./...
ok github.com/puellanivis/broken-coverage 0.001s coverage: 66.7% of statements in ./...
=== RUN TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok github.com/puellanivis/broken-coverage/package1 0.004s coverage: 33.3% of statements in ./...
? github.com/puellanivis/broken-coverage/package2 [no test files]
broken-coverage$ go tool cover -func=coverage.out
github.com/puellanivis/broken-coverage/package0.go:3: min 100.0%
github.com/puellanivis/broken-coverage/package1/package1.go:3: add 100.0%
github.com/puellanivis/broken-coverage/package2/package2.go:3: Mult 100.0%
total: (statements) 100.0%
I wrote this simple script to get average metric from coverage.out file: https://gist.github.com/sallyruthstruik/72019ba0041deaa1a2c0a817b4603403
@luklss practically everyone agrees this should be fixed, but the details of the fix are unclear. Someone needs to put in the time and effort to figure that out.
I wrote this simple script to get average metric from coverage.out file: https://gist.github.com/sallyruthstruik/72019ba0041deaa1a2c0a817b4603403
@sallyruthstruik I don't think this would work, since the problem here is that coverage.out doesn't contain information from packages with no tests.
One can use
-coverpkg=./...to get accurate results:
It seems this trick works in Go 1.19 but not in Go 1.20. I wonder what is the correct way to force the output of 0.0% in Go 1.20? (In case it is relevant, my workflow uses the legacy text format.)
One can use
-coverpkg=./...to get accurate results:It seems this trick works in Go 1.19 but not in Go 1.20. I wonder what is the correct way to force the output of
0.0%in Go 1.20? (In case it is relevant, my workflow uses the legacy text format.)
Yep.
In Go 1.19 it was even possible to use -run ^$ and record all coverage blocks in coverpkg, even if no test binaries depend on given package. Such empty blocks might be useful for custom profile merges and coverage analysis.
Example:
➜ tree .
.
├── go.mod
├── pkg1
│ └── hello.go
└── pkg2
├── greetings.go
└── greetings_test.go
2 directories, 4 files
Go 1.19
➜ go test -coverpkg ./... -coverprofile /tmp/cover.out -run ^$ ./...
? emptycover/pkg1 [no test files]
ok emptycover/pkg2 0.001s coverage: 0.0% of statements in ./... [no tests to run]
➜ go tool cover -func=/tmp/cover.out
emptycover/pkg1/hello.go:5: Hello 0.0%
emptycover/pkg2/greetings.go:5: Greetings 0.0%
total: (statements) 0.0%
Go 1.20
➜ go test -coverpkg ./... -coverprofile /tmp/cover.out -run ^$ ./...
? emptycover/pkg1 [no test files]
ok emptycover/pkg2 0.002s coverage: 0.0% of statements in ./... [no tests to run]
➜ go tool cover -func=/tmp/cover.out
emptycover/pkg2/greetings.go:5: Greetings 0.0%
total: (statements) 0.0%
Going deeper into the codebase of the language itself seems like this was never an intended feature, but more of a side-effect. Unfortunately the source code of the testing package is not straightforward, and many things break when I am trying to change it locally. The solution might be to introduce a new flag that enables this explicitly.
I wonder what is the correct way to force the output of
0.0%in Go 1.20?
PS: The best workaround (that has been mentioned by others): add dummy <foo>_test.go files with just one line package <foo>_test.
Je me demande quelle est la bonne façon de forcer la sortie de
0.0%Go 1.20 ?PS : La meilleure solution de contournement (qui a été mentionnée par d'autres) : ajoutez
<foo>_test.godes fichiers factices avec une seule lignepackage <foo>_test.
It's not very optimal but it's the only solution that seems to work for the moment. thx
You can use export GOEXPERIMENT=nocoverageredesign as a workaround in Go 1.20 until this issue is fixed as per https://github.com/golang/go/issues/58770
Change https://go.dev/cl/495452 mentions this issue: cmd/go: fix percent covered problems with -coverpkg
Change https://go.dev/cl/495446 mentions this issue: cmd/cover: add new "emit meta file" mode for packages without tests
Change https://go.dev/cl/495447 mentions this issue: cmd/go: improve handling of no-test packages for coverage
@thanm The commit d1cb5c06057ef4 cause regression which makes "go tool cover -html=cover.out -o cover.html" return an error if a package called "internal" in root module does not have test files.
This can be reproduced by running "make" on these Go repository: https://github.com/shuLhan/share/
Using go tip 1e690409206f (one commit before d1cb5c06057ef4),
(ins) 1 $ make [132/234]
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
? github.com/shuLhan/share [no test files]
...
? github.com/shuLhan/share/cmd/xtrk [no test files]
? github.com/shuLhan/share/internal/bytes [no test files]
? github.com/shuLhan/share/lib/contact [no test files]
...
ok github.com/shuLhan/share/lib/xmlrpc 1.017s coverage: 64.1% of statements
go tool cover -html=cover.out -o cover.html
go vet ./...
fieldalignment ./...
shadow ./...
revive ./...
mkdir -p _bin/
go build -ldflags "-s -w -X 'github.com/shuLhan/share.Version=v0.49.1-99-g7cf89da5'" -o _bin/ ./cmd/...
Using go tip d1cb5c06057ef4,
(ins) 1 $ make
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
? github.com/shuLhan/share [no test files]
...
github.com/shuLhan/share/cmd/totp coverage: 0.0% of statements
github.com/shuLhan/share/internal/bytes coverage: 0.0% of statements
github.com/shuLhan/share/cmd/smtpcli coverage: 0.0% of statements
...
ok github.com/shuLhan/share/lib/xmlrpc 1.017s coverage: 61.5% of statements
go tool cover -html=cover.out -o cover.html
cover: package e/internal/bytes is not in std (/home/ms/opt/go/src/e/internal/bytes)
Using go tip 638e0d36d2a6b,
23:35:29 ~/go/src/github.com/shuLhan/share
(ins) 1 $ go version
go version devel go1.22-638e0d36d2 Tue Oct 3 16:01:10 2023 +0000 linux/amd64
23:47:11 ~/go/src/github.com/shuLhan/share
(ins) 1 $ make
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
? github.com/shuLhan/share [no test files]
...
github.com/shuLhan/share/cmd/xtrk coverage: 0.0% of statements
github.com/shuLhan/share/internal/bytes coverage: 0.0% of statements
github.com/shuLhan/share/lib/contact coverage: 0.0% of statements
...
ok github.com/shuLhan/share/lib/xmlrpc 1.018s coverage: 61.5% of statements
go tool cover -html=cover.out -o cover.html
cover: no required module provides package thub.com/shuLhan/share/cmd/bcrypt; to add it:
go get thub.com/shuLhan/share/cmd/bcrypt
make: *** [Makefile:28: test] Error 1
Thanks for the report. Could you possibly open another issue for this? Thanks.
Update: I will go ahead and file an issue for this, I think I see what the problem is.