Integration and UI Testing

[rickmanelius]

What happened (or feature request):

Updated A/C - 11/27/18

• [x] DONE: Cypress NPM Install with scripts setup and tied to make.
• [x] DONE: Cypress Sample Test Local
• [ ] INPROGRESS: Cypress Sample Test CI
• [ ] NEW: Implement First Set of Tests

---

Given the amount of user interaction on the UI, we probably need a more robust approach for integration testing with sample data.

This epic includes:

• [ ] Direct integration testing between ddev-ui and ddev, with real projects. So for example, a test to create a site would actually create a site and verify that it came up correctly.
• [ ] Testing of any ddev-ui actions that change anything on the filesystem. So for example, if we take any action (like untarring an archive), test to make sure that happened on the filesystem.
• [ ] Actual UI testing.

[rickmanelius]

As part of our focus on foundational items in milestone v0.3, I think it would be wise to review how we're currently doing integration tests and re-evaluate to see if we need to take a different approach to ensure proper coverage of all functionality through v0.2 and that it will persist moving forward.

[rfay]

There are probably 3 specific areas that I think need implementation in terms of tests.

1. Direct integration testing between ddev-ui and ddev, with real projects. So for example, a test to create a site would actually create a site and verify that it came up correctly.
2. Testing of any ddev-ui actions that change anything on the filesystem. So for example, if we take any action (like untarring an archive), test to make sure that happened on the filesystem.
3. Actual UI testing. I see a number of electron testing possibilities with a google search

[andrew-c-tran]

Current Test Strategy is Mocha unit tests that leverage fixtures and mocks to simulate output from modules with controlled input. What is lacking is integration tests that actually exercise an actual CLI instance, and browser-level click testing a-la selenium.

Proposed solution:

• one suite of integration tests that runs headless between ddev-shell module and an actual instance of ddev cli. (ddev shell is the only module that directly interfaces with ddev-cli)
• removing current filesystem mocks with actual access to the test system's filesystem to confirm and verify write/read/unpack, operations
• one suite of click tests that expand on current existing unit tests that leverage the electron app and react components via Spectron. If the familiarity of Selenium is desired, we can leverage React testUtils and connect that to a selenium chrome webdriver.

Testing with Spectron allows us to test the electron specific features as well as verify the presentational flow (clicks, modals, spinner appearance, etc) so that's highly preferred. The unit tests will ensure the modules are behaving as expected, the integration will verify the contract between CLI and ddev-shell wrapper are good and have not changed/broken with later builds.

One problem that after hours of searching I still have no answer to: how the heck can we test sudo access via a test? Though this won't be an issue any longer if/when we get the hostname issue resolved.

[rfay]

There are a couple of levels of integration we need to deal with:

• Integration with ddev
• Full gui testing (click)

They can be separate approaches.

On sudo: It's possible we don't have to test it. If the test setup creates the needed hostnames in /etc/hosts, sudo will never be needed. We would rather test sudo usage of course. But ddev start would run in this situation without needing any privileges.

[dclear]

TODO: update this issue with the agreed, initial testing strategy.

[dclear]

adding @mafriend - this is an epic that contains (some) of the current priorities around testing