spin icon indicating copy to clipboard operation
spin copied to clipboard

`wasi-blobstore` draft support

Open itowlson opened this issue 9 months ago • 1 comments

This is lurching towards readiness, although it still needs a lot of tidying. E.g.

  • A bunch of stuff needs renaming! Oh goodness does it need renaming. It's embarrassing.
  • A bunch of stuff would probably benefit from splitting into modules.
  • I'm not sure how to factor-test it. I can't follow the KV tests because InstanceBuilder::build() doesn't actually produce a dispatcher object - instead the factor setup does some terrifying shenanigans to try to hook our streams up to Spin's WASI implementation, which means the dispatcher is produced at link time rather than build time.
  • Somebody needs to look at those WASI shenanigans and tell me gently "it doesn't need to be like this Ivan."
  • I'm not sure how to integration-test it. I can integration-test the files provider no worries, but not sure about Azure and S3.
  • There are several to-dos (e.g. in the file system one we need to prevent paths escaping from the "container" root path, error messages that helpfully say "oh no").

There are some underspecification issues in the current draft of wasi-blobstore: I think I've made my implementations at least consistent, but I probably need to go back over and check that too.

The implementations in this PR are:

  • Azure blob storage
  • S3-compatible, although currently it uses the Amazon client so likely bound only to the S3 implementation
  • File system, primarily for k8s persistent volumes or other attached storage

I've tested all of them but not tested them hard.

Anyway, I'm sorry this is such a beast and I'm sorry it's still messy - I wanted to do enough to have reasonable confidence in the internal provider API, and I think I'm there bar the renaming and tidying, but it means there's a lot of code, and there's a lot of scar tissue!

itowlson avatar Mar 25 '25 02:03 itowlson

Okay there is some silliness going on where it has upgraded an unrelated dependency that now requires Rust 1.83, but that makes one of the examples fling a strange new lint. I'll see if I can roll things back. Faugh.

itowlson avatar Mar 25 '25 03:03 itowlson