Make functional tests more reliable

[thc202]

Regularly failing tests:

• [x] BadSiteUnitTest: LOG_ERROR is not defined - test rewritten
• [x] BadSiteUnitTest: ZAP_HUD_FILES is not defined
• [x] BadSiteUnitTest: Expected condition failed: waiting for element
• [ ] FramesPageUnitTest
• [x] PageAlertsPageUnitTest: 100 <1> but was: <0> - hopefully fixed?

Some of the functional tests (tutorial) fail sometimes, e.g.:

org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
    org.openqa.selenium.JavascriptException: ReferenceError: LOG_ERROR is not defined
    Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
    System info: host: 'travis-job-acbae740-7b82-4d89-9d65-e682edd3a1f2', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '1.8.0_151'
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5738, moz:profile: /tmp/rust_mozprofile.RP2aKE..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
    Session ID: ad53072a-f6f6-4e5e-a008-bf11c07ff06c
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
        at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
        at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:132)

org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
    org.openqa.selenium.TimeoutException: Expected condition failed: waiting for element ([[FirefoxDriver: firefox on LINUX (66975787-1a03-4eae-b8d0-8a28b235e83f)] -> id: left-panel]) to become stale (tried for 20 second(s) with 500 milliseconds interval)
    Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
    System info: host: 'travis-job-9f26b00f-c973-416a-a13c-86c968b31ddd', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '11.0.1'
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5516, moz:profile: /tmp/rust_mozprofile.3VCarq..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
    Session ID: 66975787-1a03-4eae-b8d0-8a28b235e83f
        at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:81)
        at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:271)
        at org.zaproxy.zap.extension.hud.ui.uimap.HUD.openUrlWaitForHud(HUD.java:160)
        at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:127)

org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:35)
        at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:185)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelHidden(FramesPageUnitTest.java:147)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesHidden(FramesPageUnitTest.java:153)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanelsHiddenAndRevealed(FramesPageUnitTest.java:108)

org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest > testTaskAndNextButton(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:54)
        at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:195)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:184)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:179)
        at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:508)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest.testTaskAndNextButton(PageAlertsPageUnitTest.java:100)

(The above stack traces might not be exactly the same in develop, but the assertions should be the same.)

[thc202]

For current state: https://travis-ci.com/zaproxy/zap-hud/branches

[dscrobonia]

Thank you for adding this issue @thc202 ! Should we possiibly downgrade these 4 tests to some sort of "alpha" tests or "in development" tests until we can shake out what cuases the unreliability? Maybe still run them in Travis but don't fail on them yet?

[psiinon]

I'd rather thay stayed mandatory - we can just quick off the tests again if they fail. But def agree we want to get to the bottom of the problems and make them reliable

[dscrobonia]

Why do we want mandatory tests that aren't reliable? Am I missing something?

[psiinon]

Because if they consistently fail then we have a problem. If they fail we restart them, if they fail several times in a row then we investigate.

[dscrobonia]

That seems to defeat the purpose of the automated tests. I really don't want to have to keep restarting tests or ignoring them. I think only reliable tests should be mandatory.

[psiinon]

We'll have to agree to disagree then :P I'm all for making them more reliable, but if they become option then we'll just ignore them and never fix them. I think we need more tests, not less, so lets focus on adding more and making the ones we've got better :)

[thc202]

The first and second failures will no longer be relevant once #358 is merged (the HUD is no longer using ZAP_HUD_API).

The fourth failure should be fixed with #376.

[psiinon]

Just added a checklist to the first comment

[thc202]

Checked BadSiteUnitTest: ZAP_HUD_FILES is not defined, no longer happening in the latest builds.

[thc202]

Recent failures of the pending task:

org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
        at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:35)
        at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:137)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelVisible(FramesPageUnitTest.java:130)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesVisible(FramesPageUnitTest.java:136)

org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:35)
        at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:185)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelHidden(FramesPageUnitTest.java:151)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesHidden(FramesPageUnitTest.java:158)

[thc202]

Checked FramesPageUnitTest: 147 expected: < false > but was: < true >, per latest changes.

There are still some failures in PageAlertsPageUnitTest though.

[thc202]

One of the builds failed for FramesPageUnitTest:

org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
    org.openqa.selenium.TimeoutException: Expected condition failed: waiting for org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest$$Lambda$427/692921824@1ac39757 (tried for 10 second(s) with 500 milliseconds interval)
    Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
    System info: host: 'travis-job-2b9f0e9f-bbfe-470e-8bc8-8894104a37a7', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '1.8.0_151'
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 65.0.1, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 10883, moz:profile: /tmp/rust_mozprofile.8hFp2m..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
    Session ID: b9f866c5-8fbc-456d-8c2f-66646f701ca7
        at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:81)
        at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:271)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkWithRetry(FramesPageUnitTest.java:127)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelHidden(FramesPageUnitTest.java:147)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesHidden(FramesPageUnitTest.java:153)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanelsHiddenAndRevealed(FramesPageUnitTest.java:114)

will take a look.

[thc202]

The change in #425 addresses a failure in FramesPageUnitTest that happened when waiting for the panels to be hidden/visible (because of the overlap it could open the Alerts dialogue instead of pressing the button to hide/show the frames).