Stop loading the manifest in the orchestrator

[jnicklas]

trafficstars

The way our manifest works currently is that we generate the manifest source file, and then we compile this file using rollup. The next step is that the orchestrator imports the built manifest and adds it to its state tree. When we launch a test run, we take the manifest from the state tree, filter it as appropriate and tell the agent what the resulting tree looks like. The agent then proceeds to start running the given tree, loading the manifest in the runner frame as necessary to obtain the source code.

However this presents us with a complicated issue: since the orchestrator needs to load the manifest, the manifest must actually be loadable in both node AND the browser. This prevents us from using any browser-specific libraries which cannot be loaded in node (e.g. localforage). It also means that if we use any browser-specific API in a static context, the test will fail to load.

An alternative option would be to not load the manifest in the browser. Instead, when we start running the test, we tell the agent to load the manifest, it will fetch a copy, perform any necessary filtering, and report the resulting tree back to the orchestrator, which can incorporate the state in the state of the test run.

An advantage of this is that the agents can actually have differing manifests, which could be useful for excluding certain tests on certain browsers, or if the manifest fails to load on one browser, but not another, we could still run the tests in one browser at least.

One downside is that we no longer have a queryable representation of the manifest in our state tree, but since we don't currently have anything that relies on having this, we could simply omit this feature. If we do want to retain it, there are a few possible ways forward:

1. Automatically load any built manifest in the currently running agents, and report back the resulting state tree, maintain a per-agent manifest tree as state
2. Query the state tree from the agents on demand.

I think I like the latter option better, since it is far less resource intensive.

If we make this change it gives us a straight-forward path towards fixing #294 and #592

[dagda1]

I like option 2 but would you not notify the agents that a state change has happened.

The agents would subscribe to a particular slice.

[cowboyd]

A really easy way to approximate this would be to evaluate the test bundle in the context of JSDOM https://github.com/jsdom/jsdom#executing-scripts That would probably get us 99% of the way there, even once we've moved manifest loading completely out of the orchestrator it might be a good idea to keep it around since it would avoid having to add an extra set of protocols just to load the manifest.

GitHub

jsdom/jsdom

A JavaScript implementation of various web standards, for use with Node.js - jsdom/jsdom

[minkimcello]

@jnicklas #826 seems to be the reason the server tests are failing. Do we have plans to fix this soon?