Understand macOS test incompatibilities

[eustas]

TODO:

• [ ] fix shared library suffix on osx (ExampleFuzzerWithNative)
• [ ] get in touch with libFuzzer to see if it could have better exit code support on OSX (JpegImageParserFuzzer)
• [ ] investigate if stack overflow support on OSX could be improved (ExampleStackOverflowFuzzer)

Bazel report:

//examples:ExampleFuzzerWithNative                                       FAILED in 3.0s
//examples:ExampleStackOverflowFuzzer                                    FAILED in 2.3s
//examples:JpegImageParserFuzzer                                         FAILED in 23.0s

ExampleFuzzerWithNative and ExampleStackOverflowFuzzer fail with Unexpected exit code: 139 and no other useful output.

JpegImageParserFuzzer really starts fuzzing, discovers java.lang.NegativeArraySizeException and then fails with Unexpected exit code: 0

[fmeum]

I can reproduce all these test failures in the CI, which is why these tests are currently not run automatically for macOS.

Let me elaborate a bit on why they fail so that we can turn this issue into a tracking bug:

• ExampleFuzzerWithNative seems to fail purely because of Bazel's inability to create correctly named shared libraries on macOS (.dylib instead of .so). Fixing this is not a high priority as native library fuzzing should still work for user-supplied projects with correctly named native libraries.
• JpegImageParserFuzzer runs in libFuzzer's experimental fork mode, which uses subprocesses. Fuzzing works fine, but the main fuzzer process does not correctly pick up the exit codes of the subprocesses. Given that libFuzzer uses its own pretty complex polyfill for the system() libc function that is missing from macOS, I'm not even sure that the issue lies on our side.
• ExampleStackOverflowFuzzer is still a mystery to me, but JVMs on macOS have a history of surprising handling of StackOverflowErrors. I would really like to fix this issue as it could hint at some JNI UB in Jazzer, but at least the verbose JNI mode does not produce any findings.

[eustas]

Simply marking tests with tag does not prevent them from execution (I just run bazel test ...:all). Perhaps it is better to have something like BROKEN_ON_DARWIN_TAGS variable that is used as tags on broken tests. This variable itself should be assigned using selector and contains empty list by default, but is equal to ['manual'] if platform is darwin.

If you like, I can prepare PR for that.

[fmeum]

If you like, I can prepare PR for that.

That would be highly appreciated. Our CI pipeline is still quite simplistic and I don't have access to physical macOS devices atm.

[eustas]

Oops, not so simple. Currently "tags" is not configurable, as is was easier to implement. Now Bazel team has not enough man/will-power to fix it. The glorified "platforms" approach is not yet ready to be used,... not until Bazel 5.0 at least.

So, the only remaining solution would be to pass the parameter to fuzz_target_test_wrapper.sh. It is much less elegant, but still viable. Will try that tonight.

[fmeum-ci]

@eustas I switched to target_compatible_with to skip the failing tests. ExampleFuzzerWithNative is working on macOS now, the other two tests still fail somewhere deep in macOS internals.

[fmeum]

I changed the title of the issue to better reflect what is left to do.