WASM Parser

[jameskerr]

Update

This PR has been changed to rely on the "/compile" endpoint from the default lake which runs locally in Zui. It also changes the code so that all the information about a query is gathered once and stored in the Redux store. Components then do not need to re-parse the ast to get information about the current query.

The updates to the Zed Wasm package are still included in this PR, but zed-wasm is not used by Zui.

As a side effect, I've added a more robust mechanism for the zui-player tests to wait for the correct set of results after issuing a query. Many of our flaky integration tests were due to this problem.

Original

In an attempt to do away with having two Zed syntax parsers, we've exported the parse method from go Zed in zed-wasm.

To summarize the changes:

1. Simplify Zed Wasm Repo

• The source code is written in ESModules which can be run directly with node, no build step needed.
• The package.json type is set to module.
• We only build a commonjs version and a browser version of the package to be used by electron, jest, and all of the npm eco system that still chokes on Node ESM.
• We build a browser bundle that can be used on the web.
• We expose the main.wasm file as an export for clients to fetch and initialize on their platform. For example, in the browser we fetch(main.wasm) but in node we readFile(main.wasm).
• We return the {zq , parse} functions for the createInterface function, which is called by both entry files.
• We run playwright tests for the browser build.
• We run node tests for the esm source.
• We run jest tests for the cjs build.

I do not expose the esm versions of the files because I can't figure out how to get node to then use the cjs version if I expose both.

2. Usage in the App

• In Zui, there is a new protocol handler that responds to requests with an "app-asset://" url scheme. We can then look at the host and the path and respond with appropriate files. I use this to serve the main.wasm file from the node_modules directory.

• I mark the zed-wasm package as "external" so that it remains on the file system in a predictable place after the esbuild step bundles the code.

• I mark zed-wasm as a production dependency so that it gets included when we build the app.

fetch("app-asset://node_modules/@brimdata/zed-wasm/dist/main.wasm")

• We store the zedWasm instance in a global variable on the page.
• In the Jest tests, we initialize a zedWasm global in the before-env-setup file.

Concerns

For every test file in jest, we fetch the main.wasm file and initialize it. It's pretty slow. It's also slow when we have to initialize it on every browser window. We also have to keep a global variable around everywhere the app needs to run (tests, browser).

I have to wonder, could it be easier to expose the parser in the zq cli tool? Then the main process can parse it quickly and return the result with ipc. It would not have any startup cost and it would run quickly in tests.

This work is still good though, because we will create a playground at some point (this year I would hope) and this will certainly be used.