Experiment with fuzztest

[scpeters]

Desired behavior

Some of our existing gz-math tests are already structured like property-based tests, so convert some of them to use FuzzTest. They should still be able to work as unit tests, but when run with a fuzzing engine, they may find more edge cases instead of the current hard-coded test cases.

Alternatives considered

Implementation suggestion

For example, the following tests are candidates:

• https://github.com/gazebosim/gz-math/blob/gz-math8/src/Helpers_TEST.cc#L513-L528
• https://github.com/gazebosim/gz-math/blob/gz-math8/src/Inertial_TEST.cc#L206-L272
• https://github.com/gazebosim/gz-math/blob/gz-math8/src/Inertial_TEST.cc#L318-L371

For a first prototype, I would use a vendored copy of fuzztest (similar to our gtest_vendor folder), but we would need to decide whether we want to vendor it or not.

Additional context

[shashank1300]

Hey, I'd like to work on this.

[scpeters]

@shashank1300 great! If you haven't done fuzz testing before, can you try following the Quickstart with CMake tutorial?

From the Create and run a fuzz test, it looks like fuzz tests can be created for a void function that takes certain parameters and contains GTest expectations. Maybe the closest example that I see in gz-math is the void VerifyDiagonalMomentsAndAxes(const math::Vector3d &_moments) function in MassMatrix3_TEST.cc. If fuzztest wants native types, you could pass three doubles instead of a Vector3. One complication is that there are limits on which inputs are valid (each value must be positive, and the set must not violate the triangle inequality), so maybe this isn't the best example.

It might be easier to add simpler tests like IntegerAdditionCommute for our Vector types

[scpeters]

another starting point would be to write a function that takes two int64_t values and calls secNsecToTimePoint then timePointToSecNsec and verifies that the outputs match the inputs. Likewise with secNsecToDuration and durationToSecNsec:

https://github.com/gazebosim/gz-math/blob/gz-math8/include/gz/math/Helpers.hh#L545-L578

[shashank1300]

Hey @scpeters, I appreciate the help. I'll start right away.

[kscottz]

@shashank1300 how is it going?

[shashank1300]

@kscottz I did start it but ran into some unknown error. And I haven't been able to pick it up again since, as I am busy with my Uni exams. I'll continue after a few weeks.

[kscottz]

Let us know when you circle back. Good luck on exams!

[XINJIANGMO]

@kscottz @scpeters Hello developers ! I am interested in this too ! Today, I learn from Quickstart with CMake and complete fuzzing enging test of https://github.com/gazebosim/gz-math/blob/gz-math8/src/Helpers_TEST.cc#L513-L528.

#include <cstdint>
#include <set>
#include <tuple>
#include <iomanip>
#include <cmath>
#include <limits>
#include "gtest/gtest.h"
#include "fuzztest/fuzztest.h"
#include "gz/math/Rand.hh"
#include "gz/math/Helpers.hh"
#include <gz/utils/SuppressWarning.hh>
using namespace gz;
namespace {

void PairUnpairCorrectness(uint16_t a, uint16_t b) {
    uint16_t c, d;
    math::PairOutput key = math::Pair(a, b);
    std::tie(c, d) = math::Unpair(key);
    EXPECT_EQ(a, c) << "a: " << a << ", b: " << b;
    EXPECT_EQ(b, d) << "a: " << a << ", b: " << b;
}

void PairKeyUniqueness(const std::vector<std::pair<uint16_t, uint16_t>>& pairs) {
    std::set<math::PairOutput> keys;
    for (const auto& pair : pairs) {
        uint16_t a = pair.first;
        uint16_t b = pair.second;
        math::PairOutput key = math::Pair(a, b);
        
        EXPECT_TRUE(keys.find(key) == keys.end())
            << "Duplicate key found for pair (" << a << ", " << b << ")!";
        
        keys.insert(key);
    }
}

FUZZ_TEST(MathFuzzTests, PairUnpairCorrectness)
    .WithDomains(
        fuzztest::Arbitrary<uint16_t>(),
        fuzztest::Arbitrary<uint16_t>()
    );
FUZZ_TEST(MathFuzzTests, PairKeyUniqueness)
    .WithDomains(
        fuzztest::UniqueElementsVectorOf(
            fuzztest::PairOf(
                fuzztest::Arbitrary<uint16_t>(),
                fuzztest::Arbitrary<uint16_t>()
            )
        ).WithMaxSize(1000)
    );
} 

It can build and run successfully, and no bug found. In the process of writing , I find a point we should pay more attention(since I learn it today, please correct me if I am inaccurate):

1. Because it is a double loop in Helper_TEST.cc there are no duplicate pairs of keys, but if the PairKeyUniqueness above will appear. If it is run directly, an error will be reported, but this does not conform to the test logic, so the UniqueElementsVectorOf constraint is performed on the input field of the second function. This means that we must ensure that the input domain of the fuzz testing engine version is consistent with the original test input domain. The first test conversion is relatively simple. The second candidate will use custom type(or we can do a test transitionz, as @scpeters said above, pass three doubles and impose constraints in the input field).