Speeding up distribution tests

[rok-cesnovar]

Description

The distribution tests on develop currently run for 12h. The recently added varmat signatures basically added a x2 testing burden but the distribution tests were unsustainable even before that.

Adding beefier machines is not an option here because the machines we use are already fast.

Ideas to speedup testing:

• use precompiled headers (most of the time falls under compiling so maybe?)
• only test non-similar signatures (meaning that we only test with foo(vector) if the function supports foo(vector), foo(row_vector), foo(array[] real)

Anyone has any idea here?

Current Version:

v3.4.0

[SteveBronder]

I think pre-compiled header for basic math signatures would help. And I think when we generate the tests we could probably precompile the test fixtures. I think there are tools to see where compilation time is going, but my guess is that a lot of the time is going to be in the actual test generation where we have a 100 typedefs like

typedef boost::mpl::vector<double, double, double, empty, empty, empty> type_v_0;

That are then pumped into

typedef boost::mpl::vector<AgradDistributionVonMises, type_v_0> AgradDistributionVonMises_v_0;

INSTANTIATE_TYPED_TEST_SUITE_P(AgradDistributionVonMises_v_0, AgradDistributionTestFixture, AgradDistributionVonMises_v_0);

I think this whole thing needs a big rewrite to compile better. Like for instance in the test fixtures we have to do these double lookups at compile time like

template <class T>
class AgradCcdfLogTestFixture : public ::testing::Test {
 public:
  // gets out AgradDistributionVonMises etc.
  typename at_c<T, 0>::type TestClass;
  // gets types used in test (or empty)
  typedef typename at_c<typename at_c<T, 1>::type, 0>::type T0;
  typedef typename at_c<typename at_c<T, 1>::type, 1>::type T1;
  typedef typename at_c<typename at_c<T, 1>::type, 2>::type T2;

but if you look at how those T1, T2, etc. are used it's mostly just to define partial returns and make local versions of the calling types. But if we can get rid of a lot of that and have the generated code look something like

typedef std::tuple<AgradDistributionVonMises, double, double, double> AgradDistributionVonMises_v_0;
INSTANTIATE_TYPED_TEST_SUITE_P(AgradDistributionVonMises_v_0, AgradDistributionTestFixture, AgradDistributionVonMises_v_0);

Then I think just doing some tuple stuff we could avoid a good bit of compile time overhead. But we should probably do some sort of compiler profiling before we mark this as a thing we should refactor

[SteveBronder]

Though an easier approach here may just be on the jenkins, check if anything in /test/prob changed and if not then use a precompiled header else rebuild and stash the precompiled header

[syclik]

It definitely could use a complete rewrite. A lot of that was done because we were pre-C++11. It's a mix of code generation, template, and macros. All of it could be redone with some thought.

I'm revisiting the code now.

[rok-cesnovar]

Bumping this and assigning to myself so I dont forget. The distribution tests on develop now run for 16 hours. Really need to get this one down.

[rok-cesnovar]

A bit more on this: the trouble here is definitely the compile times, the total run time for the distribution tests is ~33 seconds. There are now over 9 million tests being run here though, spread across more than 3.5k test files.