Fuzzing halts on expected errors

[MischaU8]

When an expected error is received, the fuzzing halts. My expected behaviour is that the fuzzing will continue and ignore the error, unless it is actually not received at all.

Minimal example to reproduce this:

$ npx jazzer fuzz-expected-error-bug.js --sync -x IgnoreThisError
Dictionary: 4 entries
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4054261925
INFO: Loaded 1 modules   (512 inline 8-bit counters): 512 [0x1280c0000, 0x1280c0200),
INFO: Loaded 1 PC tables (512 PCs): 512 [0x115000000,0x115002000),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 4 ft: 4 corp: 1/1b exec/s: 0 rss: 132Mb
INFO: Received expected error "IgnoreThisError".

$

contents of fuzz-expected-error-bug.js:

"use strict"
const { FuzzedDataProvider } = require("@jazzer.js/core")

module.exports.fuzz = function(fuzzerInputData) {
    let data = new FuzzedDataProvider(fuzzerInputData)
    let number = data.consumeIntegralInRange(1, 2**35-31)

    if (number > 100) {
        throw new IgnoreThisError("Nothing to see here")
    } else if (number == 25) {
        throw Error("You found it!")
    }
}

class IgnoreThisError extends Error {
    constructor(message) {
      super(message)
      this.name = "IgnoreThisError"
    }
}

Jazzer 1.6.1 on OSX, node v18.17.1.

[bertschneider]

Hi, thx for reporting this!

expected_errors is used to influence the application exit code, so that external applications can detect expected and unexpected errors, e.g. this comes in handy in CI pipelines and scripts.

Skipping findings, as you suggested, is still on the list of features to implement. It shouldn't be too hard to build this, though. The general idea is to hash a finding's stack trace and and print that as de-duplication token on exit. In subsequent runs it can be used to ignore findings resulting in the same token/hash. Do you want to give it a try?

BTW: Nice blog post you wrote there on Jazzer.js :+1:

[harisab2547]

Yes, the Jazzer js halts on expected error but you can control by your own code to ignore those, I have made a list of errors to ignore the errors so that the Fuzzer continue without halts. If you want i can share in the thread

[karfau]

Good to know: https://www.llvm.org/docs/LibFuzzer.html#id41

It may be desirable to reject some inputs, i.e. to not add them to the corpus.

For example, when fuzzing an API consisting of parsing and other logic, one may want to allow only those inputs into the corpus that parse successfully.

If the fuzz target returns -1 on a given input, libFuzzer will not add that input top the corpus, regardless of what coverage it triggers.

[bertschneider]

True, but Jazzer.js does not support this, at least for now.

Actually, Jazzer.js uses a patched libFuzzer version where returning -2 from the fuzz target will exit the fuzzing loop so that the application can be shut down orderly. The result of the JS fuzz target is not evaluated on the C++ side. Looking up usages of RETURN_CONTINUE and RETURN_EXIT (https://github.com/CodeIntelligenceTesting/jazzer.js/blob/main/packages/fuzzer/shared/libfuzzer.h#L30) should probably point to all required places, though.

Happy to help you implement this feature :smile:

[karfau]

Well, that's good to know. Maybe it's worth documenting the differences compared to the official docs on libFuzzer side?

It also took me a while to find the documentation for the output, and I would have loved to pass -ignore_c_rashes flag when running with -fork, but I'm told that the flag is not supported...

I guess there are more differences, and a quick overview about those would be helpful to understand what is possible and what is not.

PS: Even if I would have the time to look into it, I have no clue regarding C/C++ so I don't feel capable of diving into the code base to implement this.

[bertschneider]

No worries, thx for bringing this up! I added it to our list of missing features.

Ideally users would not need to know which or even if an external fuzzing engine is used internally. As libFuzzer has quite a few useful features not directly exposed by Jazzer.js it's possible to pass in fuzzer engine arguments directly. And you're definitely right that documentation is lacking in this regard.

To enable fork mode and ignore crashes you could use something like this: npx jazzer <fuzz_target> -- -fork=1 -ignore_crashes=1

In general libFuzzer parameters can be passed in after --. Please note that libFuzzer parameters use a single -, whereas Jazzer.js parameters use a --.

[karfau]

Ah, I was missing the =1 part when passing the argument! Already did gave uptrying it, that's great news.