Rewrite TestBenchCommandExecutor::waitForVaadin to check whether dev server is ready

[denis-anisimov]

Problem: the master Flow branch will use another way to run a web applications. Previously (p)npm install and webpack dev server were started as a part of ServletContainerInitializer ( DevModeInitializer ) and they blocked the whole web app startup.

Now they won't block anymore ServletContainerInitializer but will be executed in a separate thread. It means that the servlet container is able to start much faster and web app will be ready to respond to requests very soon. But it also means that ITs won't work because if Dev mode server is not started yet then a web page which reloads itself will be shown. This page will open the proper Vaadin app page once dev mode server is started (after a number of reloads). But ITs expects that app is ready right away after the server startup . This expectation is not true anymore.

It means that we either need to fix ITs (that's made in the proper PR) or fix is generally via TB.

Here is the template HTML which is used to show the page if dev server is not yet ready: https://github.com/vaadin/flow/pull/7633/files#diff-81b2106e76906ae6b77f7a23f19c0c5e

As you can see window.Vaadin.Flow.clients is explicitly set to null which should allow to return false here:

var clients = window.Vaadin.Flow.clients;"
            + "if (clients) {"
            + "  for (var client in clients) {"
            + "    if (clients[client].isActive()) {"
            + "      return false;"
            + "    }"
            + "  }"
            + "  return true;"
            + "} else {" +
            // A Vaadin connector was found so this is most likely a Vaadin
            // application. Keep waiting.
            "  return false;"
            + "}";

( a part of WAIT_FOR_VAADIN_SCRIPT ). So it means once the page is reloaded and Vaadin app is ready waitForVaadin should stop. It stops but it doesn't allow to get a page of the real Vaadin app after reload. Once waitForVaadin it's still possible to see the "waiting page".

So it's either related to wrong handling location.reload(); JS or selenium/Chrome driver bug or whatever.

What I did is: modified waitForVaadin to

Object result = ((JavascriptExecutor) getDriver()
                    .getWrappedDriver()).executeScript(
                            "return window.Vaadin && window.Vaadin.Flow && window.Vaadin.Flow.devServerIsNotLoaded;");
            finished = !Boolean.TRUE.equals(result)
                    && (Boolean) ((JavascriptExecutor) getDriver()
                            .getWrappedDriver())
                                    .executeScript(WAIT_FOR_VAADIN_SCRIPT);
            if (finished == null) {

and this code does what it should. After it's finished the real Vaadin page is correctly loaded. This code is safe to add because it's backward compatible: pages doesn't have window.Vaadin.Flow.devServerIsNotLoaded and it will be undefined (false ) which won't affect anyhow the old ITs.

I suggest to modify TB code in this way.

But please feel free to modify the template in Flow whatever you want to fix the behavior if you don't want to touch TB. I just don't know why it doesn't work out of the box.

[denis-anisimov]

@alvarezguille , please take a look.

[Artur-]

Running a start.vaadin.com test on Vaadin 16 sometimes fail with the following screenshot.

Isn't this a really fundamental problem that will break all tests for everybody?

[Artur-]

I can reproduce this on a Mac by

1. Taking the default starter application
2. Wait for output to say "Starting webpack-dev-server"
3. Press ctrl-s to stop the console output, which at the same time stops webpack from finishing
4. Run a test

What happens is that the first lookup in the test waits for max 20s and then proceeds. This is will fail quite often as on my machine, a quite new Mac, webpack takes around 20s to finish. Presumably if you have a little bit older machine, it will fail every time

[Artur-]

Changing the hardcoded 20000 in https://github.com/vaadin/testbench/blob/master/vaadin-testbench-core/src/main/java/com/vaadin/testbench/commands/TestBenchCommandExecutor.java#L121 to 60000 seemingly resolved the problem for me

[Artur-]

Maybe a better solution would be to wait a long time for the "frontend dev build has not yet finished" and not change the default timeout