Grab `.wasm` binaries from the web

[ecumene]

I want to be able to share modules with others but not require them to download/compile them themselves

Spin.toml

My intuition says this URL would be written in the spin.toml like so:

[[component]]
id = "key-value"
source = "https://github.com/ecumene/components/.../hello-world.wasm"

But how would we get the modules? I think there are generally two ways to achieve this

1. Build a tool that saves them to a cache based on the spin.toml and change nothing here
2. Have Spin download these at runtime

We could also do both! Maybe we could include a mapping of these modules to their place on the drive, and that would leave it up to something built into Spin or some manager to resolve them.

[Mossaka]

We could combine the two ways: having spin download these wasm files at runtime and cache them to a spin.lock file, which includes the checksums. In this way when the next time spin is running, if the checksum of the spin.toml is not changed, then can use the cached wasm files. And of course if the lock file is deleted, then spin will always fetch the newest remote wasm modules.

[ecumene]

Would the lockfile map the downloaded modules to their place on disk too?

[radu-matei]

I would decouple the two conversations as two distinct ones:

1. using HTTP URLs as module sources in spin.toml files
2. caching pulled modules (and assets), which also includes when running from Bindle, and I think it makes sense to have a common caching implementation for various sources.

For 1, do we need to consider any authentication mechanisms when performing HTTP requests, or are we ok with public sources for now? (I think for the first iteration on this, public sources like GitHub releases are totally fine).

[ecumene]

For 1, do we need to consider any authentication mechanisms when performing HTTP requests, or are we ok with public sources for now?

Like HTTP basic auth? Or something else like a cookie?

For my purposes public sources are fine

[Mossaka]

Yep I agree that this is two tasks. I'm happy to work on the caching implementaiton.

[lann]

I think we should default to verifying the integrity of remote-sourced binaries to encourage good security practices. That might look something like:

[component.source]
url = "https://raw.githubusercontent.com/..."
integrity = "sha256:abc123..."

If necessary, we could allow explicit integrity opt-out, e.g.: integrity = "insecure"

This could also (later) be expanded nicely to code signing, e.g. integrity = "ed25519:<pubkey>".

[itowlson]

I'm not opposed to this (provided we address the software supply chain risks) but would like to get a sense for where it fits in with bindle publishing and the registry. Presumably the situation here is that a developer would like to reuse a handler, such as Bartholomew, but within their own application context; whereas bindles are more about providing full applications.

Should we instead provide a convenient way for users to publish and reference module-only bindles, with all the guarantees that provides consumers around provenance and immutability? Or would the "plain HTTP" solution still be necessary because there aren't enough well known bindle servers around?

Longer term this speaks to questions such as the component registry, and the composing of applications from public, published components. So let's get a sense of where we envisage this being used, whether it is about convenient short-term sharing within a close collaboration, or long-term publishing of components designed for reuse.

I guess I am worried that we risk doing the unreliable, physically coupled publishing strategy first, and risk ending up with a spaghetti ecosystem with high vulnerability to supply chain attacks, which becomes entrenched before we have a structured alternative. On the other hand, I do see the value of convenient sharing and don't want to hold it up while we labour on a 'perfect' solution!

[lann]

As long as the remote item is integrity-checked, it should be no less secure than wgeting the module and referencing that local file.

Most of the value of bindle (or a future registry) is what it can (could) add on top: component discovery, code signing key management, update management, dependency resolution, etc.