cclib
Implement standards for test jobs
I think we should come up with some Markdown notes that formalize the process of creating and running test inputs, and in the process also implement some standards that each test job should follow. This is sort of a catch-all for things we currently do (DVB, B3LYP/STO-3G, using a particular set of coordinates, etc.) and things we would like to change. Perhaps creating the notes should be a separate issue, since that doesn't imply changing how calculations are run.
A lot of things fall under this category, but for example, two in particular come to mind:
- B3LYP as the density functional. Many programs let you choose the definition, but this isn't always true, and sometimes this is very confusing; see http://chemistry.stackexchange.com/questions/39411/definition-of-the-b3lyp-functional-in-common-qc-programs. This is even ignoring which form of VWN is chosen, which is what is usually meant by "definition of B3LYP".
- Symmetry on by default. Again, because symmetry is implemented to different levels in different programs, this can make comparing against reference data very difficult. Programs reorient molecules to different positions (center of mass, center of nuclear charge, alignment on specific axes), which greatly affects orientation-dependent properties (polarizability, quadrupole components). Strange (false) results can also sometimes appear when a certain property isn't implemented properly for symmetry, which is more common than one expects.
My proposed solutions:
- Switch to a density functional with an unambiguous definition. My vote is for PBE0. It is in every package and there is only one definition for the correlation and exchange functionals. I also think we can avoid many grid issues (always a problem when doing DFT comparisons) in our results by keeping with a hybrid functional, rather than say PBE.
- Disable all symmetry. The alternative is to switch to a molecule that doesn't have symmetry, but it is probably impossible to rerun some of the old unit tests, and since most people run calculations with symmetry on, we can always have regressions for when symmetry issues arise.
Other things are DFT integration grid quality, SCF convergence thresholds, and other less fundamental things that I'll leave open-ended for now.
One side-effect of making these changes is that many of the special-case tests, particularly those that do comparisons with reference data, might be made obsolete. I am of course willing to re-run all the calculations with the programs I have available to me.
I think this is a great idea, and it might be a good chance to move some of the docs into the main repo. The docs repository should really only generate the published HTML, we should keep the soruces for the docs here if someone wants to read them locally.
I think your specific suggestions are fine. Will comment more when I have a chance to think about this a bit longer.
I'm now thinking that with 2.x we might want to clean things up a bit repository-wise. The unit tests have grown substantially over the years and there's no reason why they won't continue. So, perhaps we should think about creating three separate new repositories when jumping to version 2.x:
- cclib for the code and docs
- cclib-data for the unit test data
- cclib-regressions for the regression data