FLiT icon indicating copy to clipboard operation
FLiT copied to clipboard

Allow multiple test points in a single test case

Open mikebentley15 opened this issue 7 years ago • 0 comments

Currently, you can return a single string or floating-point result at the end of each test case. This is analogous to a unit testing framework that only allows a single assertion at the end in the form of a single Boolean return value.

This enhancement opportunity is analogous to having multiple assertions in a test case. The idea is to allow to have multiple "comparison points" similar to assertions. Something like the following:

virtual void
run_impl(const flit::TestInput<T>& ti, const flit::GroundTruth<T>& gt) override {
  T a = ti.vals[0], b = ti.vals[1], c = ti.vals[2];
  flit::COMPARE("first", a, gt, [](T x, T y) { return x - y; });
  flit::COMPARE("second", b, gt, [](T x, T y) { return std::abs(x - y); });
  flit::COMPARE("third", c, gt, [](T x, T y) { FLIT_UNUSED(y); return x; });
  flit::COMPARE("fourth-string", "some-string", gt, [](std::string x, std::string y) {
    return x.size() - y.size();
    });
}

Then this would result in four entries into the database for each time this test is executed. Each of these entries is a "test point" (similar to an assertion) that is named by the first argument to flit::COMPARE. The calls to flit::COMPARE would replace the return value. Also the compare() method in the class would be removed and instead function pointers (or lambda expressions) would be used in each flit::COMPARE call. Supported types would be the same: float, double, long double, and std::string.

This is at least the idea. What do others think?

mikebentley15 avatar Jul 05 '17 17:07 mikebentley15