pFUnit

pFUnit copied to clipboard

I am attempting to convert a test to pFUnit.

I have something like this

  call test_set_quadrature_exponential(1.0_dp, tolerance)
  call test_set_quadrature_exponential(2.0_dp, tolerance)

where test_set_quadrature_exponential has the asserts.

this approach I don't think works with pFUnit because the subroutine test_set_quadrature_exponential wouldn't have the @test keyboard and therefore will not get parsed?

I was trying to follow the parameterized test cases but it looks like a lot more mental load to wrap my head around. it appears to use an object oriented approach.

My interpretation is that one has to write a class that extends the ParameterizedTestCase, and is decorated with @TestCase and the additional testParameters, and a constructor? (is this where the magic happens?)

and also write a class that extends the AbstractTestParameter decorated with @testParameter that specify the parameters (what defines the parameterized variables) that can get stuffed into an array that is passed to the @TestCase decorator attribute testParameters

@TestCase decorator attribute Constructor creates a new instance for each parameter set in the parameters array?

The actual test under the @test decorator creates an instance of the extended ParameterizedTestCase class and then calls the SUT functionality?

Where is the magic that is iterating through the parameterized values? Is there a simpler example out there for each of these features?

or is this the best way to do it?

The @test and other annotations are only for user convenience. (Often a very big convenience.) But you may also just want to wrap your test within a pFunit test:

@test
subroutine wrap_test()
   ...
  call test_set_quadrature_exponential(1.0_dp, tolerance)
  call test_set_quadrature_exponential(2.0_dp, tolerance)
end subroutine wrap_test()

(Or maybe put each such call inside its own wrapper.) You can then make calls to assertEqual() within your tests that will be caught and reported by the framework. Lots of little caveats of course. If you want file and line number you'll need to explicitly put that on your call to assertEqual. Likewise, if you want the test to end you'll need to add a call to see if any exceptions have been put on the stack at each level so that you can return early. Happy to have a more detailed discussion if this is the path you want to take.

Or ... ParameterizedTestCase: ParameterizedTestCase is something you only want to take on if you have several tests that all need to loop over a consistent set of parameters. And even then duplication might be easier to implement (but probably harder to maintain.) Looping a parallel test over a varying number of processes is the most common example, and pFUnit provides specialized support for that case to keep it relatively simple. (MpiTestMethod)

It's been a while since I have implemented a ParameterizedTest case, so I had to go back and review the code to recall how it all ties together. In the end, each parameterized test is given just one parameter. Something at a higher level must then construct a different test object for each desired parameter. The easiest way to see this is to look at the output from an existing use case after the pfunit preprocessor has filled in the boiler plate. The end of this process is a module that wraps the user-provided module to provide a custom constructor and then an external function that constructs a test suite. I can provide the full before and after if you are still interested. But the bits you would need to implement would look something like below:

module WrapTest_LatLon_GridFactory
   use FUnit
   use Test_LatLon_GridFactory
   implicit none
   private

   public :: WrapUserTestCase
   public :: makeCustomTest
   type, extends(Test_LatLonGridFactory) :: WrapUserTestCase
      procedure(userTestMethod), nopass, pointer :: testMethodPtr
   contains
      procedure :: runMethod
   end type WrapUserTestCase

   abstract interface
     subroutine userTestMethod(this)
        use Test_LatLon_GridFactory
        class (Test_LatLonGridFactory), intent(inout) :: this
     end subroutine userTestMethod
   end interface

contains

   subroutine runMethod(this)
      class (WrapUserTestCase), intent(inout) :: this

      call this%testMethodPtr(this)
   end subroutine runMethod

   function makeCustomTest(methodName, testMethod, testParameter) result(aTest)
      type (WrapUserTestCase) :: aTest
      character(len=*), intent(in) :: methodName
      procedure(userTestMethod) :: testMethod
      type (GridCase), intent(in) :: testParameter
      aTest%Test_LatLonGridFactory = Test_LatLonGridFactory(testParameter)

      aTest%testMethodPtr => testMethod
#ifdef INTEL_13
      p => aTest
      call p%setName(methodName)
#else
      call aTest%setName(methodName)
#endif
      call aTest%setTestParameter(testParameter)
   end function makeCustomTest

end module WrapTest_LatLon_GridFactory

function Test_LatLon_GridFactory_suite() result(suite)
   use FUnit
   use Test_LatLon_GridFactory
   use WrapTest_LatLon_GridFactory
   implicit none
   type (TestSuite) :: suite

   class (Test), allocatable :: t

   type (GridCase), allocatable :: testParameters(:)
   type (GridCase) :: testParameter
   integer :: iParam 
   integer, allocatable :: cases(:) 
 
   suite = TestSuite('Test_LatLon_GridFactory_suite')

   testParameters = getParameters()

   do iParam = 1, size(testParameters)
      testParameter = testParameters(iParam)
   call suite%addTest(makeCustomTest('test_shape', test_shape, testParameter))
   end do

   testParameters = getParameters()

   do iParam = 1, size(testParameters)
      testParameter = testParameters(iParam)
   call suite%addTest(makeCustomTest('test_centers', test_centers, testParameter))
   end do


end function Test_LatLon_GridFactory_suite

thank you @tclune, that is exactly the information I think I needed.

For now I am pushing toward the first suggestion, and as you mentioned I need to add information about the line number manually? In the example below call test_set_quadrature_exponential(0.1_dp, 1.0e-16_dp) fails. The error lists the line number of the failing assert but doesn't track which call to the function it failed on, so this is where I would need to add something of a stacktrace and line number information? is that syntax that the @test decorator can handle or is it manual?

I agree with what your are saying about the parameterized test, I was leaning towards duplication since it only covers two sets of parameters. ParameterizedTestCase looks great! for large sweeps, but a potentially massive hammer unless I know there will be more parameter sets added. I will very likely get to a point that I will implement some test cases that use this feature, but not yet.

Test code:

  @test
  subroutine wrap_test_set_quadrature_exponential()
    real(dp) :: tolerance
    ! Exponential tests
    tolerance = 1.0e-5_dp
    call test_set_quadrature_exponential(1.0_dp, tolerance)
    call test_set_quadrature_exponential(2.0_dp, tolerance)
    ! added the following to get a failure example
    call test_set_quadrature_exponential(0.1_dp, 1.0e-16_dp)
  end subroutine wrap_test_set_quadrature_exponential

  subroutine test_set_quadrature_exponential(alpha, tol)
    use mathlib, only: set_gauss_laguerre_quadrature
    real(dp), intent(in) :: alpha
    real(dp), intent(in) :: tol
    real(dp), dimension(32) :: points
    real(dp), dimension(32) :: weights
    real(dp) :: approx
    integer(ikind) :: Nmax
    call set_gauss_laguerre_quadrature(Nmax, points, weights)
    points(1:10) = points(1:10)/alpha

    ! Let <f(x)> = \int_{0}^{\infty} f(x) \alpha * \exp(-\alpha x) dx
    ! Check <1> = 1
    approx = sum(weights(1:Nmax))
    @assertEqual(1.0_dp, approx, tolerance=tol)
    ...

Regarding which call failed, you have a few options. One would be to simply use a separate wrapper for each of the three calls. A bit tedious, but mostly cut-and-paste. Then the name of the wrapper will unambiguously identify which call failed. The other is to add a string argument to the inner procedures and pass a different string each time. Could even be 'A', 'B', and 'C' but something more informative would be useful. And then the call to assert has an optional argument message where you could use that string. I use that latter technique frequently where I have simple "parameterized tests":

@test
subroutine test_foo()

    call check(1.0, 2, 'case 2')
    call check(2.0, 3, 'case 3')
    call check(7.0, 5, 'case 5')

contains
   subroutine check(expected, arg, description)
       real, intent(in) :: expected,
       integer, intent(in) :: arg
       character(*), intent(in) :: description

       @assertEqual(expected, f(arg), message=description)
   end subroutine check

end subroutine test_foo

Thank you @tclune that was exactly what i needed to know about the message attribute.

Is there a good place to add an example like this in here or in the demos?

I think adding it to the demos would be best. Just add a new subdirectory. The hard part will be choosing a meaningful name for the directory. Probably should reserve "Parameterized" for more complex example that exercises the heavy machinery.