Multi-Wasm Test Harness

[dOrgJelli]

Is your feature request related to a problem? Please describe. Currently we do not have a clean way of testing multiple wasm languages, and forming a feature matrix. We now support AssemblyScript and Rust, and with TinyGo support on the way very soon it would be nice to have a system created for this.

Describe the solution you'd like See @cbrzn comment below for implementation details.

Additional context Currently we have test wrappers that are duplicated in each language here: https://github.com/polywrap/toolchain/tree/origin/packages/test-cases/cases/wrappers

Once these issues has been closed, this issue can be considered done:

• [x] https://github.com/polywrap/wasm-test-harness/issues/1
• [x] https://github.com/polywrap/wasm-test-harness/issues/2
• [ ] https://github.com/polywrap/wasm-test-harness/issues/3
• [ ] Remove test-cases/wasm from current codebase

[cbrzn]

I have been thinking about this and my first thoughts are:

The different WASM implementations are composed of different parts that allow us to create a WASM file (aka wrap.wasm) that is compatible with WRAP protocol, a high level ok the flow to generate this file is:

• Schema.grapqhl & implementation (index.ts or lib.rs) are given
• An ABI is generated (which includes the entire schema package but let's not talk about this), allowing us to create generated files (called bindings)
• The wrap.wasm is generated with the WRAP exports through the build process (currently using docker https://github.com/polywrap/toolchain/tree/origin/packages/cli/src/lib/defaults/build-images/wasm)

Based on this flow, I would suggest a gameplan that would look something like:

• Move the test-cases/cases/wrappers folder outside of the toolchain repository, making it a standalone repository
• The structure of the tests would change a bit, to not repeat the folders between every language, but rather, we would have a folder for every test case, and inside of this we can have an implementations folder with subfolders for every lang? Something like:

|- test-cases/
  |- asyncify/
    |- schema.graphql
    |- polywrap.yaml
    |- implementations/
      |- rs/
        |- Cargo.toml
        |- lib.rs
      |- as/
        |- package.json
        |- index.ts
  |- simple-storage/
    |- schema.graphql
    |- polywrap.yaml
    |- implementations/
      |- rs/
        |- lib.rs
        |- Cargo.toml
      |- as/
        |- index.ts
        |- package.json

The polywrap.yaml would be somehow a template that would have different values based on the project type (rs|go|as)

• Use the current client (JS) to test the different generated wasm files in every test case, the same way we are currently doing, but having the opinionated stuff in one folder (implementations), making the development experience better when it comes to adding a new WASM implementation to test. For example, this PR would be simple with this approach

It's worth mentioning that the main pain point on this issue (at least from my perspective) is that for every test case, we need to give an opinionated input per WASM implementation (example: index.ts|lib.rs or Cargo.toml|package.json). We can probably have a default dependencies file and add it in the test case only if we want to customize it.

Also, we can have some statistics to see which tests fail in which implementation, allowing us to implement a tool that creates an explicit report of what features a WASM implementation currently supports. I envision it to be something like the Hive UI, where we can see which client (in Ethereum context) ran and which tests they passed. Imagine having a UI where we can see which tests the WASM implementation are ok/failed, all of this using the JS Client at first; eventually, when we also create a test harness for clients we can have the same UI to show tests per client implementation, and also being able to mix clients & wasm implementations.

Hopefully, this makes sense. Open to any suggestions or ideas; just trying to create something useful here :smile:

[cbrzn]

Another important point is that currently, we have these tests in the wasm-as.spec.ts and wasm-rs.spec.ts files - We may definitely not rely on jest for test and instead migrate to do all the tests with polywrap.test.yml to make integrations with workflows.

Eventually, it would also be possible to run yarn polywrap run and add an argument where we pass a binary of a client? Making it easier to do the multi-client & multi-wasm test-harness possible (But this is another topic :yum:)

[nerfZael]

The approach makes sense from an organizational perspective: It's easy to keep track of features, and each feature can have implementations in different languages. I think it's good to have a shared schema and (partial) manifest for all languages of a feature, but we might have language-specific things we add to the schema and manifest in the future (to change the way the bindings are generated), so I'm unsure how future-proof that is. I like the idea of a UI, but we can also just have a CLI-based UI (table) to show off the features/languages.

And yes, we should definitely move the test-cases repo out of the monorepo.

[dOrgJelli]

Sounds like a great plan @cbrzn, I'm all for it!

[cbrzn]

This being developed here https://github.com/polywrap/wasm-test-harness

[dOrgJelli]

@cbrzn what requirements would we need to meet for this task to be completed? Some thoughts:

• All wasm tests are runnable in the new repo
• Feature matrix can be automatically produced
• Addition of new languages is "easy" (can test with TinyGo)
• test-case wrappers in the toolchain repo have been removed (or use the wrappers from the new repo if needed)

[cbrzn]

Yes, that would be a perfect list of tasks, will add those into the description of the issue