feat: Add NATS blobstore provider

[ossfellow]

This commit adds a new blobstore provider that uses NATS JetStream as its backend storage. The provider implements the wasi:blobstore/blobstore interface and includes:

• Access to the full range of NATS Object Store Storage features, including persistent and ephemeral storage, and self-expiring objects
• Configurable write timeout to accommodate different computing environments
• Comprehensive integration tests using testcontainers
• Updated http-blobstore example to demonstrate usage of all the blobstore features

The nats-blobstore-provider codebase has been broken down into multiple files for improved maintenance, and to help new contributors understand the structure and the constituent parts of a wasmCloud provider. It also makes it easy to understand which WASI interfaces are implemented by the provider.

Feature or Problem

Implementing WASI blobstore proposal by taking advantage of NATS capabilities.

Related Issues

Addresses #3262

Release Information

next release

Consumer Impact

None

Testing

The provider includes unit tests, for configurations, and a comprehensive integration test suite. However, the integration tests are not running successfully, most likely because of setup issues; all tests have the ignore flag set to prevent any adverse effects; so, this should not prevent merging the PR.

To compensate, the http-blobstore rust example component has been completely overhauled to test all WASI blobstore functions, which has been used to validate the NATS blobstore provider functionality as well.

It will provide great value to identify the setup issues in the integration test suite, for future use cases, outside the this PR lifecycle.

Unit Test(s)

All passed

Acceptance or Integration

Please see Testing.

Manual Verification

Using the http-blobstore example component

[ossfellow]

@brooksmtownsend, thank you for taking the time to review the PR, and all the great feedback provided. I'd like to mention a few things for your consideration:

1. I used the Azure blobstore provider - which seemed to be the most up to date - as the basis of the implementation techniques and approach used; however, the NATS blobstore provider has more than a dozen improvements in areas of logic, efficiency and performance over that making it our flagship blobstore provider, and the basis for improving the other 3 blobstore providers.
2. I took your recommendation and implemented concurrent object deletion in batches of 50; however, I'm not sure if it's a good idea to make it user configurable - similar to the max write buffer for optimized operations in different environments - or it would be an overkill. If you think making it configurable provides a meaningful performance advantage, I'll add that, and I'm open to do it as part of this PR (i.e. tonight), or later, if we're trying to clear the PR backlog before the holidays.
3. The integration test suite is also based on Azure's; but it also has a lot of improvements and documentation; however, as reported before, the wRPC calls fail. If you look at the link setting, you'd see that I set the wrpc-blobstore/blobstore interface, but the error messages, which I shared on our Slack channel allude to wasi-blobstore/blobstore being served here; this might point to an issue in our testing framework.

[brooksmtownsend]

Hey @ossfellow just a quick ask here, would you mind rebasing this PR off of main to help get rid of the commits that are from upstream and not a part of this PR? Let me know if this is complicated and if we can help

[ossfellow]

Hey @ossfellow just a quick ask here, would you mind rebasing this PR off of main to help get rid of the commits that are from upstream and not a part of this PR? Let me know if this is complicated and if we can help

@brooksmtownsend , done; please let me know if anything's amiss

[brooksmtownsend]

I do reproduce the test failures on debug mode 😢

[github-actions[bot]]

This PR is now formatted in line with Conventional Commits

[ossfellow]

@thomastaylor312, @brooksmtownsend, in the latest round of updates, in addition to what I mentioned in PR feedback responses, I took the hint @Mossaka had provided and treat deleting non-existent objects gracefully. I also added a new integration test for it.

[ossfellow]

Smallest change, I know that this will trigger spelling errors in the docs build unless it is marked as code.

PR updated.

[thomastaylor312]

Sorry for the delay here @ossfellow. A rebase and you should be good to go

[ossfellow]

@thomastaylor312, @brooksmtownsend, thank you for the reviews and help. I've rebased and squashed to commits into one.

[thomastaylor312]

I think a cargo check --all should update the lock file and solve the issue here

[ossfellow]

@thomastaylor312, thank you for the tip; it's done; the async_nats API has changed, and added headers and metadata, which I've added, without supplying any values.

[lachieh]

🎉 Thanks to everyone involved with getting this across the line!