add fuzz testing

[lemire]

For increased reliability, we should add extensive fuzz testing.

[richardstartin]

This might be easier to do properly with Java 8 support.

[lemire]

I think we could drop support for Java 7 in main releases.

[lemire]

@richardstartin Can we close this?

[richardstartin]

I'd leave it open. There are things that aren't fuzz tested yet (most obviously anything involving mutated bitmaps).

[lemire]

Leaving open.

[richardstartin]

These tests catch errors, and trigger assertions in the code, but don't make it easy to capture a reproducer. I suggest we serialise the bitmap triggering an error, base64 encode it and save it to a JSON file describing the error.

[larsk-db]

Sorry to necro this ancient issue, but I'm trying to extend the fuzz testing support to Roaring64Bitmap.

Sadly I'm literally failing at step 0: How do I run the existing fuzz tests?

I did some googling on how to use bloody gradle, and it suggested that something like the following should do the trick:

./gradlew -Porg.gradle.project.roaringbitmap.fuzz-tests=true test

But apparently that isn't the case :(

Anyone know what the magic invocation is @richardstartin @lemire ?

[richardstartin]

It looks like they have rotted because they don't get run very much (way too costly to run in CI), it's been a couple of years since I've run them, in 5 minutes of trying I couldn't get them to run :(

It's definitely worthwhile resurrecting (or reworking) them because they found a lot of issues in the 32 bit implementations and I suspect lots of issues will turn up for Roaring64Bitmap.

[larsk-db]

It looks like they have rotted because they don't get run very much (way too costly to run in CI), it's been a couple of years since I've run them, in 5 minutes of trying I couldn't get them to run :(

Alright, thanks for checking and confirming I'm neither insane nor dumb :D

It's definitely worthwhile resurrecting (or reworking) them because they found a lot of issues in the 32 bit implementations and I suspect lots of issues will turn up for Roaring64Bitmap.

I concur. Roaring64Bitmap has been having so many issues recently which people randomly found, that I really feel we need this to have any confidence in the implementation at this point.

[richardstartin]

Roaring64Bitmap has been having so many issues recently which people randomly found, that I really feel we need this to have any confidence in the implementation at this point.

I started looking at this myself just to demonstrate there's a problem and that we should probably include a warning that Roaring64Bitmap is currently beta quality, but I ran into memory consumption issues quickly and I consider memory parsimony a prerequisite for reliable fuzz/property testing. Some of the problems can be resolved easily, but one of the biggest problems, that the containers aren't embedded in the leaf nodes of the ART, and instead have a 64 bit "pointer" into a paginated array of containers can't be changed without breaking serialisation compatibility, so I gave up at that point.

[larsk-db]

I started looking at this myself just to demonstrate there's a problem and that we should probably include a warning that Roaring64Bitmap is currently beta quality, but I ran into memory consumption issues quickly and I consider memory parsimony a prerequisite for reliable fuzz/property testing. Some of the problems can be resolved easily, but one of the biggest problems, that the containers aren't embedded in the leaf nodes of the ART, and instead have a 64 bit "pointer" into a paginated array of containers can't be changed without breaking serialisation compatibility, so I gave up at that point.

Do you have a pointer to a branch I could use as a starting point? No need to redo all the work you already put in.

[richardstartin]

I got sidetracked by the OOMEs pretty quickly and don't have anything worth sharing. (I do have a branch where I removed the NodeType enum which reduces the size of some ART nodes (alignment prevents this from being universally effective) which I can push if there is interest.)