nengo-gui
nengo-gui copied to clipboard
nengo
Unit Tests
Making unit tests with Selenium is really easy (good docs here and here). I'm not saying we should test every feature, but a few quick sanity tests would be nice.
Off the top of my head:
- to see if we've broken any graphs
- check if iPython hasn't exploded
- check if the basic network generation (does entering new code generate new items?) is functional.
But before I get ahead of myself listing a bunch of things to test, we need to agree on how we're going to do this. @pblouw and @arvoelke you were apparently working on this before the summer school? Did you use Selenium?
We had a number of tests running in Selenium covering basic things like making graphs, starting and stopping a simulation, changing input values, etc. The tests are almost certainly broken now due to changes to the gui, but its been on my todo list to update them and get them working properly again. Suggestions for a list of features to test would be welcome.
On Tue, Jul 7, 2015 at 12:54 PM, Sean Aubin [email protected] wrote:
Making unit tests with Selenium is really easy (good docs here https://selenium-python.readthedocs.org/en/latest/index.html and here https://selenium.googlecode.com/svn/trunk/docs/api/py/index.html). I'm not saying we should test every feature, but a few quick sanity tests would be nice.
Off the top of my head:
- to see if we've broken any graphs
- check if iPython hasn't exploded
- check if the basic network generation (does entering new code generate new items?) is functional.
But before I get ahead of myself listing a bunch of things to test, we need to agree on how we're going to do this. @pblouw https://github.com/pblouw and @arvoelke https://github.com/arvoelke you were apparently working on this before the summer school? Did you use Selenium?
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471.
Where can I find these old tests?
On Tue, Jul 7, 2015 at 2:16 PM, pblouw [email protected] wrote:
We had a number of tests running in Selenium covering basic things like making graphs, starting and stopping a simulation, changing input values, etc. The tests are almost certainly broken now due to changes to the gui, but its been on my todo list to update them and get them working properly again. Suggestions for a list of features to test would be welcome.
On Tue, Jul 7, 2015 at 12:54 PM, Sean Aubin [email protected] wrote:
Making unit tests with Selenium is really easy (good docs here https://selenium-python.readthedocs.org/en/latest/index.html and here https://selenium.googlecode.com/svn/trunk/docs/api/py/index.html). I'm not saying we should test every feature, but a few quick sanity tests would be nice.
Off the top of my head:
- to see if we've broken any graphs
- check if iPython hasn't exploded
- check if the basic network generation (does entering new code generate new items?) is functional.
But before I get ahead of myself listing a bunch of things to test, we need to agree on how we're going to do this. @pblouw https://github.com/pblouw and @arvoelke https://github.com/arvoelke you were apparently working on this before the summer school? Did you use Selenium?
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471.
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471#issuecomment-119290004.
I don't think I pushed them to github, so you'll have to come talk to me.
On Tue, Jul 7, 2015 at 2:18 PM, Sean Aubin [email protected] wrote:
Where can I find these old tests?
On Tue, Jul 7, 2015 at 2:16 PM, pblouw [email protected] wrote:
We had a number of tests running in Selenium covering basic things like making graphs, starting and stopping a simulation, changing input values, etc. The tests are almost certainly broken now due to changes to the gui, but its been on my todo list to update them and get them working properly again. Suggestions for a list of features to test would be welcome.
On Tue, Jul 7, 2015 at 12:54 PM, Sean Aubin [email protected] wrote:
Making unit tests with Selenium is really easy (good docs here https://selenium-python.readthedocs.org/en/latest/index.html and here https://selenium.googlecode.com/svn/trunk/docs/api/py/index.html). I'm not saying we should test every feature, but a few quick sanity tests would be nice.
Off the top of my head:
- to see if we've broken any graphs
- check if iPython hasn't exploded
- check if the basic network generation (does entering new code generate new items?) is functional.
But before I get ahead of myself listing a bunch of things to test, we need to agree on how we're going to do this. @pblouw https://github.com/pblouw and @arvoelke <https://github.com/arvoelke
you were apparently working on this before the summer school? Did you use Selenium?
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471.
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471#issuecomment-119290004.
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471#issuecomment-119290332.
Having gotten the tests from Peter, @tbekolay have you looked into running this stuff with TravisCI? It seems to be non-trivial.
It definitely takes some doin'. I can help with setting up the .travis.yml, but this seems to be a good place to start. Basically, TravisCI comes with Firefox and a way to get a virtual GUI environment, so it should be possible to test with Selenium on TravisCI (for the Firefox driver, in any case).
Yeah, I did see that, however that appears to be for FireFox only. Additionally, for other web-browsers it seems the only way to (easily) integrate it with TravisCI is to use a commercial server from Sauce. I guess it's not impossible to run our own server, but this seems kind of like the same problem we ran into when we thought about testing our builds on Windows. Thoughts?
On Tue, Jul 7, 2015 at 2:53 PM, Trevor Bekolay [email protected] wrote:
It definitely takes some doin'. I can help with setting up the .travis.yml, but this http://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-GUI-(e.g.-a-Web-browser) seems to be a good place to start. Basically, TravisCI comes with Firefox and a way to get a virtual GUI environment, so it should be possible to test with Selenium on TravisCI (for the Firefox driver, in any case).
— Reply to this email directly or view it on GitHub https://github.com/nengo/nengo_gui/issues/471#issuecomment-119302405.
Sauce Labs cost $$$ so probably not worth it. Testing through TravisCI on Firefox will definitely catch many issues that we wouldn't catch otherwise; we will miss a few things here and there not testing in Chrome and Safari, but it's not worth all of the extra effort to test in Chrome for those last 5% of issues when testing in Firefox will get 95% of them.
And, maybe someone will end up making a free alternative for that, like they did on Windows with AppVeyor.
A further detail that I'm a bit confused about is if we should be running JavaScript specific tests with Phantom.js and some JavaScript testing framework. I have zero experience in the domain and my Googling isn't telling me much, other than Phantom.js is used for basic functionality testing. Does anyone have experience on when/how to use Phantom.js?
Additionally, to get started configuring this repository for testing with TravisCI, I would need to have admin rights. @tcstewar how would you feel about temporarily giving me admin rights? @tbekolay says that all you would need to do is add me to the team nengo-gui-maintainers
A further detail that I'm a bit confused about is if we should be running JavaScript specific tests with Phantom.js and some JavaScript testing framework. I have zero experience in the domain and my Googling isn't telling me much, other than Phantom.js is used for basic functionality testing. Does anyone have experience on when/how to use Phantom.js?
Never used it, but my guess is that the vast majority of the stuff on the JS side of nengo_gui is interface-related, rather than basic functionality. I could maybe see testing some bits of the DataStore class that way, but that's about all that's jumping out. So my guess is that phantom.js isn't all that useful for our testing. That said, I've never actually used it so if anyone has any actual experience that should override my guess here. :)
@tbekolay says that all you would need to do is add me to the team nengo-gui-maintainers
Sure! You're on the list now.
Now that I've got a few tests running with the Nengo GUI, it's time to scale up. This is how I'm envisioning structuring the tests.
Additionally, in terms of pytest structure, I'd like each function to still be a test, but only a new instance of the GUI should be started if you use a new class.
Opinions?
Structure seems good, though just in terms of where the folder should go, the general one should be in nengo_gui/tests and the component ones should be in nengo_gui/components/tests and not in nengo_gui/tests/components. Every Python project I've looked at does it in the former way, so we should be consistent with that.
I'm not familiar enough with the implementation of things to comment on reusing the same Nengo GUI instance within tests in the same class... seems fine, as long as everyone's aware that that's what happens!
It might be good to have some more structure in the general tests folder. I suppose we won't just test for bugs, but make sure some general functionality not necessarily tied to one specific component works (e.g., that the border of an object stays below the mouse cursor while resizing).
It's apparently quite possible to run Chrome tests in TravisCI, according to this SO question and this blog post.