pints icon indicating copy to clipboard operation
pints copied to clipboard

Create a PR template for introduction of new sampling methods into PINTS

Open ben18785 opened this issue 4 years ago • 3 comments

ben18785 avatar Feb 02 '21 13:02 ben18785

Good description of how to do so here: https://blog.axosoft.com/enhancing-pull-request-descriptions-templates/

ben18785 avatar Feb 02 '21 13:02 ben18785

As part of this, I'm thinking that people need to specify hyperparameters for their method (in a file on PINTS somewhere) that are hand-picked for each of the problems we use for functional testing. I think we probably already have something like this for existing methods on the original functional testing repo...

It might also be worth having a "score sheet" for a method presented before it's merged: showing its performance on the functional tests. Whether we'd want such a score sheet in PINTS: I doubt it. But, we could have a separate repo with scores for each method.

ben18785 avatar Feb 02 '21 15:02 ben18785

We're thinking it should have:

  • optimal hyperparameters for each functional test
  • average (I think mean) score across each of the functional tests

ben18785 avatar Feb 09 '21 09:02 ben18785