docs(sip): add `spin test` SIP

[radu-matei]

Signed-off-by: Radu Matei [email protected]

[itowlson]

I know you had already decide to rethink this so forgive me if I'm jumping the gun. From your description this morning, I think this is a really useful and valuable idea, and Spin's clear decoupling of triggers, services and user code means we can do some great stuff in this space. So I wanted to think out loud about some possible scenarios and approaches.

I feel like we need to be more flexible than mock request-expected response fixtures. Examples:

• A case where the response is semi-deterministic. The developer wants to check that the response is a 200 status code and contains the string All glory to the Hypnotoad, but (say) the response also contains the current date, and it doesn't matter if the string appears in upper or lower case.

• A trigger doesn't return a response at all, except possibly an error (e.g. Redis or cron), but the developer wants to test the side effects (e.g. record written to log table, body stored to object store). Indeed, they may want to check this even where the trigger does return a response.

The first scenario suggests we want to be able to inject a request and just receive the response, rather than having spin test check it. This allows the user's test framework to perform whatever assertions it wants, and to report failures in whatever way it wants. A possible way to accomplish this is for spin test to take an event message on the command line, and return the response via stdout/stderr (and then exit immediately).

Obviously, the downside of this is that it means the developer needs to run some kind of testing tool, and needs to set up the infra to run an external binary from that tool, rather than just going spin test. So it may be that what I'm describing is a different use case from what you want spin test to address.

The second scenario suggests we will need convenient ways to fake out host ~services~ components. Lann's work to inject these at top level gives us a really easy and elegant way to do this - we just need to do test implementations of our outbound interfaces. And for many of these we already have "zero standup" implementations, e.g. local object store; we could easily do one for outbound Redis I assume. (Not sure about outbound HTTP but maybe that doesn't need to be faked because you don't need to stand anything up for it anyway!) So spin test could just wire up these instead of the default implementations, allowing test authors to set up fake data and examine side effects.

Again, apologies if I'm jumping in before you're ready - I'm more excited by this feature than I really ought to be. Hopefully these thoughts will be useful as you refine this.

[lann]

A case where the response is semi-deterministic.

You know, it would be very nice if the test environment could be made fully deterministic, and it might even be feasible, at least in a lot of cases. Wasmtime's WASI impl can be configured with custom clocks and RNGs...

The second scenario suggests we will need convenient ways to fake out host services components

This was a big part of my motivation for being able to replace host components with Wasm components:

# profile.test.toml (?)
[[component]]
id = "outbound-http"
source = "mock-outbound-http.wasm"
files = { source = "dog-facts-test-data.json", dest = "/data.json" }

[radu-matei]

This PR hasn't had any updates in a while. Closing for now, but definitely open to revisiting a testing framework for Spin applications in the future.