axe-selenium-python
axe-selenium-python copied to clipboard
mozilla-services
Tests to verify the sample code in the readme functions
I'm not sure if this is possible but wanted to capture the intention after seeing issue #149 filed by @mudgil. Can a process, like those run in TravisCI, verify when a pull request is made that sample code embedded in the Readme continues to function?
I recognize this might not be possible, or that the effort involved may be too great. Let's do a brief investigation into the viability of this.
@m8ttyB I was wondering the same thing. updating the readme and examples is something that I often forget when making changes...
@davehunt Do you know of any tools that can lint or test example code in docs?
There's doctest but that's based on Python's docstrings. I'm not aware of anything that would be able to test example code from a README though. Another option might be to build the README from a snippet that's tested independently.
Another option might be to build the README from a snippet that's tested independently.
That sounds promising. After doing a quick search, there may be tools that exist already to test code in readme files. Will follow up on this.
@kimberlythegeek @m8ttyB I second what @davehunt said about testing a code snippet independently. It is possible to write a script that reads the README, extracts the python code from it and then runs it through a linter like flake8 or pycodestyle. I'd be happy to give this a shot.
Before the code in the file can be checked for errors, we first need to extract or find it in the document.
There are two options I can think of to solve this:
- To convert the README from
.rstto.mdformat. It is relatively simple to extract code out of markdown files. Converting from one format to another will require using a third party tool for converting documents like pandoc. - Second option is to work on the README directly to extract the python code from it using regular expressions.
I prefer option 2. I am working on figuring out how to write the regex pattern that'll work in this case.
@m8ttyB @kimberlythegeek Update: I have written a script that extracts Python code from a README document and runs checks on it using flake8. I put up this gist .
I am not entirely sure what the best way to invoke tests from the script is, but this proves at least that linting code in READMEs can be done.
This might be worth a look at, it is a tool for linting Python code blocks in markdown files: https://github.com/johnfraney/flake8-markdown,