conventionalcommits.org
conventionalcommits.org copied to clipboard
conventional-commits
Conformance test cases
I'm maintaining a parser but having to rely on hope that I read and interpreted the spec correctly (for example, BREAKING-CHANGE is in all caps but it seems like it could be lower case).
It would be helpful to have a set of conformance tests.
One idea is a data file that has a list of commit messages and parsed fields (with None for missing optional fields). Each language's parser can then just embed this in their tests and iterate over it, parsing and comparing the results with the data file. We'd need some error cases as well but I don't think we can easily specify "expected" results for errors other than "should fail".
@epage I like this idea 👍 and I think it fits with the direction I'd love to see us going in.
I am using the following tests in my go library:
https://github.com/zbindenren/cc/blob/main/tests.yaml
Not sure if yaml is the best format. Maybe we could agree to something.
Maybe not suited for everyone, but what I've done is I've taken the bigger libraries that use CC (most are from the JS ecosystem) and wrote a custom script that saves every commit message into a text file. I then parse them with my parser and save the result in some kind of meta format which makes it super easy to compare (similar to the yaml one).
For parsing operations that fail I take a closer look as to why. Maybe the commit message is just not in the CC format. Helped me out a lot to find some bugs.
I'd then go through the messages, see if the parsed result is the expecting one when compared to the original text file and if that is the case, I'll add it to my test cases.
It's not the best idea, but it definitely helps when something breaks due to a rewrite or special case.
Popular alternatives could be property-based testing where you randomly generate CC compliant messages and see if they parse correctly. You'd have to write the generation rules yourself though that opens up a potential place for bugs. But it's a viable solution if the generation rules are solid.
@SirWindfield thats a great approach for helping with regressions; my concern when raising this issue is with ambiguities or misreadings of the spec. Having a common set of conformance tests / properties would help make sure we are all on the same page.
Maybe we should start by making a list of what kind of commit messages the test suite should contain. Should we create one commit message per possible combination? Like no_body_no_scope_no_footer, no_body_scope_no_footer etc. ?