Future of testing metadata

[berquist]

As mentioned in https://github.com/cclib/cclib/pull/506, testing metadata will become difficult as more keys become added, since little is standardized and nothing is documented yet.

These are the keys currently used for metadata that are tested in some way:

• basis_set
• input_file_contents
• input_file_name
• methods
• package
• package_version

These are untested keys:

• coords
• functional
• info
• keywords
• success
• unrestricted
• warnings

All mentions of metadata in the test directory:

test/regression.py:        assert self.data.metadata["symmetry_full"] == "c1"
test/regression.py:        assert self.data.metadata["symmetry_abelian"] == "c1"
test/regression.py:        assert self.data.metadata["symmetry_full"] == "c2h"
test/regression.py:        assert self.data.metadata["symmetry_abelian"] == "c1"
test/data/testGeoOpt.py:        self.assertTrue(self.data.metadata['success'])
test/data/testSP.py:    @skipForParser('ADF', 'Does not support metadata yet')
test/data/testSP.py:    @skipForParser('GAMESSUK', 'Does not support metadata yet')
test/data/testSP.py:    def testmetadata(self):
test/data/testSP.py:        """Does metadata have expected keys and values?"""
test/data/testSP.py:        self.assertTrue(hasattr(self.data, "metadata"))
test/data/testSP.py:            self.assertIn("basis_set", self.data.metadata)
test/data/testSP.py:            self.assertIn("input_file_name", self.data.metadata)
test/data/testSP.py:            self.assertIn("input_file_contents", self.data.metadata)
test/data/testSP.py:        self.assertIn("methods", self.data.metadata)
test/data/testSP.py:        self.assertIn("package", self.data.metadata)
test/data/testSP.py:        self.assertIn("package_version", self.data.metadata)

(I'm not sure how the ones for symmetry made it in to regressions.)

This issue is to collect thoughts about how to structure tests for metadata, which will make documenting it easier in the future, and also what additional things should go in metadata in the near future. I can poke in https://github.com/cclib/cclib/issues/451, which is directly related.

[berquist]

https://github.com/cclib/cclib/pull/599 tried this by making metadata its own calculation type, only for the purposes of testing. What we really need is to be able to test metadata across different calculation types, or having something like #737.