OCaml AFL fuzzing examples

Small examples of how to use AFL to fuzz OCaml programs

These examples are here to help you quickly get set up with AFL and to illustrate an upcoming article on the Tarides blog.

Setup

To be able to correctly run the examples in this repo and toy around with fuzzing you will need to install afl and have a +afl opam switch so that the binaries are properly instrumented for fuzzing.

You can setup the afl switch by running:

$ opam switch create fuzzing-switch 4.07.1+afl

You can either install AFL from your distribution, e.g. on Debian:

$ apt update && apt install afl

Or by using the convenience opam package:

$ opam install --switch=fuzzing-switch afl

Some of the examples have extra dependencies such as crowbar and bun. You can install all of them by running:

$ opam install --switch=fuzzing-switch crowbar bun

Simple parser

The simple-parser folder contains the most basic example and shows how you can use afl-fuzz to fuzz a simple parsing function written in OCaml.

The lib subfolder contains a library with a single parse_int function that parses an int from a string, with a little twist.

The fuzz subfolder contains the code to be compiled to the fuzzing binary fuzz_me.exe which must be passed to afl and an inputs/ folder with a couple starting test cases.

You can try fuzzing it by yourself:

$ dune build simple-parser/fuzz/fuzz_me.exe
$ afl-fuzz -i simple-parser/fuzz/inputs -o _build/default/simple-parser/fuzz/findings _build/default/simple-parser/fuzz/fuzz_me.exe @@

Or simply run:

$ dune build @simple-parser/fuzz --no-buffer

which will do pretty much exactly the above.

AFL should find the crash fairly quickly. It will show up in the top right corner of afl-fuzz's output, under uniq crashes, see the picture below.

You can inspect the input that triggered the crash by running:

$ cat _build/default/simple-parser/fuzz/findings/crashes/id*
abc

and reproduce it by running:

$ cd _build/default/simple-parser/fuzz
$ ./fuzz_me.exe findings/crashes/id*
Fatal error: exception Failure("secret crash")

Awesome list

The awesome-list folder contains an example of how AFL can be used conjointly with the crowbar library to both find crashes and do some property based testing.

The lib subfolder contains a library with a single sort function that sorts lists of integers. Again it mostly works fine except in two specific cases where it will either crash or sort the list in reverse order.

The fuzz subfolder contains the code for the fuzzing binary. It is slightly different from the previous example as here we use crowbar to build the correct binary instead of doing it by hand. Furthermore, we don't check for crashes only anymore but also want to know if the function under test invariant, i.e. the resulting list is sorted in increasing order, stands. The resulting fuzzing binary has roughly two modes of execution: an AFL one and a QuickCheck one.

In QuickCheck mode it uses OCaml's randonmess source to try a fixed number of inputs. To try that mode you can run:

$ dune exec awesome-list/fuzz/fuzz_me.exe

Alternatively you can also run the QuickCheck mode until a test failure is discovered with the -i option of the binary like this:

$ dune exec -- awesome-list/fuzz/fuzz_me.exe -i

In AFL mode, it just use the input supplied by AFL as a source of randomness to supply values of the right form to your test functions. You can run it just like with a regular fuzzing binary:

$ dune build awesome-list/fuzz/fuzz_me.exe
$ afl-fuzz -i awesome-list/fuzz/inputs -o _build/default/awesome-list/fuzz/findings ./_build/default/awesome-list/fuzz/fuzz_me.exe @@

Or use the convenience dune alias:

$ dune build @awesome-list/fuzz --no-buffer

Both modes should find the bugs in a split second. In QuickCheck mode it'll pretty print the input value that triggered the failure. In AFL mode you can proceed as in the above example, i.e. kill the afl-fuzz process once it found the two unique crashes. From there, inspecting the input files won't tell you much as it's just used to seed crowbar PRNG but you can run the fuzz binary on those and the input values will be pretty printed the same way they are in QuickCheck mode:

$ cd _build/default/awesome-list/fuzz
$ ./fuzz_me.exe findings/crashes/<input_file>
Awesome_list.sort: ....
Awesome_list.sort: FAIL

When given the input:

    [4; 5; 6]

the test failed:

    check false

Fatal error: exception Crowbar.TestFailure

Bun and fuzz testing in CI

This repo also contain Drone CI scripts that run fuzz testing for both our examples. afl-fuzz is not very CI friendly by essence so we use bun. bun is a CLI wrapper for afl-fuzz, written in OCaml. It takes care of a few things for you, such as running several fuzzing processes in parallel while correctly setting afl-fuzz to do so but most of all it wraps each of those processes so that the execution fail whenever one of them finds a crash. Finally, it will also display the input that lead to the test failure so that you can try to reproduce and debug it locally.

The CI script itself is in the .drone.yml file and it looks like that:

kind: pipeline
name: amd

platform:
  os: linux
  arch: amd64

steps:
- name: build
  image: ocaml/opam2:4.07
  commands:
  - sudo apt-get update && sudo apt-get -y install afl
  - sudo chown -R opam .
  - git -C /home/opam/opam-repository pull origin && opam update
  - opam switch 4.07+afl
  - opam depext crowbar bun
  - opam install -y crowbar bun
  - opam exec -- dune build @bun-fuzz --no-buffer

As you can see, there's nothing special here. We install the dependencies and build the bun-fuzz alias.

There's one for simple-parser and one for awesome-list. The aliases are identical and defined as:

(alias
 (name bun-fuzz)
 (locks %{project_root}/bun)
 (deps
  (:exe fuzz_me.exe)
  (source_tree input))
 (action
  (run bun --input inputs --output findings -- ./%{exe})))

The default bun invocation is very similar to a regular afl-fuzz invocation. You'll note the use of (locks %{project_root}/bun) to prevent concurrent execution of fuzz tests by dune. This is required because by default bun will use all available cores and afl-fuzz won't use a core that's already running an afl-fuzz process.

You may try it locally:

$ dune build @awesome-list/bun-fuzz
09:05.39:Fuzzers launched: [1 (pid=7357); 2 (pid=7358); 3 (pid=7359);
                            4 (pid=7360); 5 (pid=7361); 6 (pid=7362);
                            7 (pid=7363); 8 (pid=7364)].
09:05.39:Fuzzer 1 (pid=7357) finished
Crashes found! Take a look; copy/paste to save for reproduction:
echo J3JhaWl0IA== | base64 -d > crash_0.$(date -u +%s)
09:05.39:[ERROR]All fuzzers finished, but some crashes were found!

or you can take a look at the latest build.

bun comes with a bunch of configuration CLI options and I invite you to take a look at its documentation by running bun --help to find out how to make the most out of it in your particular use case.

One useful bun feature is its no-kill mode. There's a bun-fuzz-no-kill alias available for awesome-list fuzz tests, it's defined as:

(alias
 (name bun-fuzz-no-kill)
 (locks %{project_root}/bun)
 (deps
  (:exe fuzz_me.exe)
  (source_tree input))
 (action
  (run timeout --preserve-status 1m bun --no-kill --input inputs --output
    findings -- ./%{exe})))

The --no-kill option tells bun to let the fuzzing processes run even after they found their first crash. That can be convenient if you're not very confident about an implementation you're fuzzing and you are expecting to find several bugs in it. What makes this mode so nice is that it integrates with timeout fairly well. When receiving SIGTERM, bun will shutdown all the afl-fuzz processes and will pretty print the crash triggering inputs to the standard output in the same way it does in regular mode which makes this a viable option for running fuzz tests in CI as well.