Support Async specs and BeforeAndAfter* hooks

[kwinter]

this allows implementing specs to use either sync or async suites

[lightbend-cla-validator]

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

[kwinter]

wanted to get this up for discussion - this should allow users to make use of either sync or async tests, but there is one notable change in behavior:

Using testOnly **ChromeExampleSpec* as an example (without the chrome driver set up) Previously, a failure to start up web driver for the Browser line would report as a cancelled test and the test run would succeed, such as

[info] Run completed in 3 seconds, 775 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 5, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0, Canceled 5
[success] Total time: 4 s, completed Apr 25, 2020 3:08:35 AM

with this change, it would show up as an aborted suite and the test run would fail:

[info] ScalaTest
[info] Run completed in 3 seconds, 751 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[error] Error: Total 1, Failed 0, Errors 1, Passed 0
[error] Error during tests:
[error] 	org.scalatestplus.play.examples.guice.onebrowserpersuite.ChromeExampleSpec
[error] (scalatestplus-play / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 9 s, completed Apr 25, 2020 3:23:56 AM

This can be observed on the Travis CI builds for the PR - the Safari/IE tests were previously showing up as canceled due to being unable to create a webdriver - but are now aborted and failing the build (tests previously not working, but are now aborted->failure rather than cancelled->success)

There could be different mindsets on whether that's a good or bad thing - but it is a change in behavior, so looking for feedback on whether that's acceptable before continuing on. If so, I think there could be a follow up to switch PerSuites over to use BeforeAndAfterAll, to address #111

[raboof]

I like the change, perhaps we should add a switch to disable tests that rely on webdriver so they can be explicitly cancelled when running on a system that doesn't have the browsers installed?

[kwinter]

that could work. also another option: it feels a bit hacky, but overriding testDataFor will allow the cancellation to keep the previous behavior

abstract override def testDataFor(testName: String, theConfigMap: ConfigMap = ConfigMap.empty): TestData = {
     webDriver match {
      case UnavailableDriver(ex, errorMessage) =>
        ex match {
          case Some(e) => cancel(errorMessage, e)
          case None    => cancel(errorMessage)
        }
      case _ => super.testDataFor(testName, theConfigMap)
    }
  }

happy to either:

• revert the browser specs back, so that at least the base app/server line is decoupled from sync
• put the testDataFor override in, to maintain behavior for the browser specs

for now, any preference?

[lightbend-cla-validator]

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

[kwinter]

for now, I've reverted the browser specs back, and added a round 2 on the app/server specs. so this PR in total, for app/server fixtures:

• allows use of sync or async (#112)
• uses BeforeAndAfterEach for PerTest and BeforeAndAfterAll for PerSuite for lifecycle start/stop
  • defers creation of app/server unless the tests will actually execute (so won't happen if the suite is excluded via tags)
  • throws exception rather than creating app/server "outside" of a test/suite execution
  • allows specs to hook onto to BeforeAndAfterEachTestData/BeforeAndAfterAll to access server/app for setup/cleanup (#111 and #121 )
• aborts the full suite for PerTest if the app/server can't be created, rather than trying once per test and failing each individually (default behavior of BeforeAndAfterAll)
• shares a single app/server if mixed in with ParallelTestExecution (previously it would create a new one each time)

not sure what to do about the mima binary change errors

[kwinter]

another issue popped up - this currently breaks the previous behavior for using ParallelTestExecution with PerSuite, like extends OneAppPerSuite with ParallelTestExecution

before: results in an app/server being created per test (perhaps surprising behavior, was to us) after: results in an exception/null

what's the intended behavior?

[resolved] - updated so that extends OneAppPerSuite with ParallelTestExecution results in sharing one single instance across all tests, and added specs for parallel execution

[lightbend-cla-validator]

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

[lightbend-cla-validator]

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

[lightbend-cla-validator]

At least one pull request committer is not linked to a user. See https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user#commits-are-not-linked-to-any-user

[SoerenSilkjaer]

Not being allowed to use either sync or async suites has bothered me for so long. Oh I would so love to see this change outthere.

[kwinter]

I've signed the CLA (a few times) - don't know why it hasn't picked it up

If someone can lend some guidance on what to do with the MIMA errors, would be happy to fix them up

[raboof]

I've signed the CLA (a few times) - don't know why it hasn't picked it up

The signature for github user 'kwinter' was indeed recorded, but the commits have '[email protected]' as the email address for the Author file, and that email address isn't associated with your GitHub account. You can associate additional email addresses as documented at https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-your-github-user-account/adding-an-email-address-to-your-github-account

[raboof]

If someone can lend some guidance on what to do with the MIMA errors, would be happy to fix them up

Hmm, that looks pretty tricky: AFAICS these changes are binary incompatible "by design", the whole point of the PR is to no longer extend the things that were extended before.

I wonder what binary compatibility guarantees we have promised for this project though - we tend to be a bit less strict in projects that are only used in test scope. Perhaps someone more intimately familiar with the project can say something about that?

[palanga]

Any updates on this ? I need it !

[kwinter]

I've associated the commit email with my account - so that's hopefully good to go

I think that just leaves the binary compatibility issue from MIMA, and guidance on what we want to do there