Support for Go 1.18 fuzz testing

[abhinav]

What version of rules_go are you using?

v0.30.0

What version of gazelle are you using?

6bbfc47f1b0a27ee1efeddcc6671f3e4e03235dc

What version of Bazel are you using?

% bazel version
Build label: 5.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Jan 19 14:08:54 2022 (1642601334)
Build timestamp: 1642601334
Build timestamp as int: 1642601334

Does this issue reproduce with the latest releases of all the above?

Yes.

What operating system and processor architecture are you using?

% uname -sm
Linux x86_64

and

% uname -sm
Darwin arm64

What did you do?

Wrote a test with a simple fuzz-test.

package fuzzing

import (
        "bytes"
        "encoding/hex"
        "testing"
)

func FuzzHex(f *testing.F) {
        for _, seed := range [][]byte{{}, {0}, {9}, {0xa}, {0xf}, {1, 2, 3, 4}} {
                f.Add(seed)
        }
        f.Fuzz(func(t *testing.T, in []byte) {
                enc := hex.EncodeToString(in)
                out, err := hex.DecodeString(enc)
                if err != nil {
                        t.Fatalf("%v: decode: %v", in, err)
                }
                if !bytes.Equal(in, out) {
                        t.Fatalf("%v: not equal after round trip: %v", in, out)
                }
        })
}

I ran it with bazel run :go_default_test. This failed because of golang/go#51623. After patching the fix for that in,

% bazel run :go_default_test -- -test.fuzz .
[...]
testing: -test.fuzzcachedir must be set if -test.fuzz is set
[...]

So I added the fuzzcachedir argument:

% bazel run :go_default_test -- -test.fuzz . -test.fuzzcachedir $(pwd)/testdata
[...]
-----------------------------------------------------------------------------
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
fuzz: elapsed: 0s, testing seed corpus: 0/6 completed
[..]

But wait, it's running without coverage guidance so I added --collect_code_coverage to instrument the binary but it still had the same issue:

% bazel run --collect_code_coverage :go_default_test -- -test.fuzz . -test.fuzzcachedir $(pwd)/testdata
[..]
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
fuzz: elapsed: 0s, testing seed corpus: 0/6 completed
[..]

What did you expect to see?

Coverage-guided fuzzing.

What did you see instead?

Less efficient fuzzing.

[sluongng]

Could you try to compile the test binary using 'bazel coverage'?

If it does not work, we might want to create a special go_binary option to build fuzz targets 🤔

[prestonvanloon]

Could you try to compile the test binary using 'bazel coverage'?

If it does not work, we might want to create a special go_binary option to build fuzz targets thinking

I can confirm that the warning message about coverage still appears when running with bazel coverage.

bazel coverage //path/to:go_default_test --test_filter=Fuzz --test_arg=-test.fuzz=Fuzz --test_arg=-test.fuzzcachedir=/tmp/fuzz --test_output=streamed

warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient

Edit:

rules_go @ v0.31.0 gazelle @ v0.25.0

[farhaven]

I don't see that warning here, and fuzzing works, with the command line from @prestonvanloon 's last comment. I've also got --@io_bazel_rules_go//go/config:debug in my .bazelrc.user for build, but that shouldn't make a difference.

rules_go @ v0.31.0 Go toolchain @ v1.18.1 Platform is arm64 macos

EDIT: I just re-checked the output of bazel coverage and the warning is indeed there.

[apex-edison-moreland]

Just piling on here, I'm running into the same problem and haven't been able to find a workaround

[srosenberg]

Adding gc_goopts = ["-d=libfuzzer"] to your go_test rule ensures (edge) coverage is generated during compilation and subsequently used during fuzzing [1],

bazel run pkg/util:util_test -- -test.run notests -test.fuzz FuzzHex  -test.fuzzcachedir /tmp/foobar/ -test.v

Executing tests from //pkg/util:util_test
-----------------------------------------------------------------------------
=== FUZZ  FuzzHex
fuzz: elapsed: 0s, gathering baseline coverage: 0/12 completed
fuzz: elapsed: 0s, gathering baseline coverage: 12/12 completed, now fuzzing with 24 workers
fuzz: elapsed: 3s, execs: 2525728 (841880/sec), new interesting: 0 (total: 12)
fuzz: elapsed: 6s, execs: 5150923 (874883/sec), new interesting: 0 (total: 12)
fuzz: elapsed: 9s, execs: 7817339 (888760/sec), new interesting: 0 (total: 12)
...

[1] https://github.com/golang/go/blob/master/src/internal/fuzz/counters_supported.go#L15

[stdll00]

gc_goopts = ["-d=libfuzzer"] option seems no effects for other rules. Set the option to the all go_binary and go_test is possible solution but it's better to specify string_list_flag like below.

bazel run --@io_bazel_rules_go//go/config:gc_goopts="-d=libfuzzer" pkg/util:util_test -- -test.run notests -test.fuzz FuzzHex  -test.fuzzcachedir /tmp/foobar/ -test.v