Expose component model in C api

[lanfeust69]

The goal is to be able to host wasm components in a wasmtime runtime embedded through the C api.

This was discussed in issue #8036.

Thanks to @rockwotj for the basis of the initial commit, in #7801.

I didn't see tests specific to the c-api, currently only tested through the example (and in our internal use !).

I'm not sure it the sizeable component.rs file should be split...

[rockwotj]

Nice work! BTW one of the more complex things I had punted on in my POC was handling resources. From my conversations with @alexcrichton those significantly changed the Rust API when adding them, and might do the same in the C API (although maybe this has changed in the time since I did the POC?). I notice there is no resource support in this PR - it's probably worth a plan for that. At least that was the sentiment prior.

[lanfeust69]

Nice work! BTW one of the more complex things I had punted on in my POC was handling resources. From my conversations with @alexcrichton those significantly changed the Rust API when adding them, and might do the same in the C API (although maybe this has changed in the time since I did the POC?). I notice there is no resource support in this PR - it's probably worth a plan for that. At least that was the sentiment prior.

Indeed, having started from your change, and not needing resources on my side, I decided not to support that for now, but I should have made that clear.

I'm currently looking at the CI failures :

• I've been "fooled" by clang-format, because there is no .clang-format file in the repo, so my default one would be picked up, with a significantly different formatting, and I assumed there was no strict formatting rules for C files.
• Not sure if a function only used with some features should be marked #[allow(dead_code)] (which may miss the time it is no longer used) or #[cfg(any(feature = "feature1", feature = "feature2"))] (which gets unwieldy, and may prevent discoverability if useful elsewhere)
• CI (reasonably) wants to run the C example for component, but the workaround in component/main.rs is not available, something else is needed.

[dicej]

Regarding resources: we'll definitely need to support them eventually, given that they're already used heavily by WASIp2 and beyond; plus they're just really useful in general. This PR doesn't need to address that, but we should start thinking about what a follow-up PR might look like.

A minimal implementation would include:

• A way to add new host-implemented resource types to a LinkerInstance, equivalent to https://docs.rs/wasmtime/latest/wasmtime/component/struct.LinkerInstance.html#method.resource
• A C equivalent to (https://docs.rs/wasmtime/latest/wasmtime/component/struct.ResourceTable.html) for allocating host resource reps (as uint32_t) and pairing them with resource instances (as void*, which may be cast to the expected type; maybe paired with a type ID for runtime sanity checking)

Otherwise, I think it's just a matter of passing the uint32_t reps (representing either host-implemented or guest-implemented resource instances) back and forth between the host and guest, letting Wasmtime handle the details of lifting and lowering. @alexcrichton does that sound plausible to you?

[fitzgen]

Not sure if a function only used with some features should be marked #[allow(dead_code)] (which may miss the time it is no longer used) or #[cfg(any(feature = "feature1", feature = "feature2"))] (which gets unwieldy, and may prevent discoverability if useful elsewhere)

We generally prefer the cfg-based approach, even though it can get unweildy, because it provides better compile times and code sizes. Discoverability is not an issue because we build docs for docs.rs with feature(doc_auto_cfg) which automatically adds "this method depends on blah feature" to all cfged functions.

[lanfeust69]

@alexcrichton : Thanks for a very quick first look !

I personally prefer sizeable PR broken down through meaningful commits (because of the logical coupling, an oversight in an earlier PR may only be really visible when used later on), but unfortunately GitHub (as well as GitLab) make the review process much more PR/MR oriented than commit-oriented, which kind of defeats this approach. I'll try to see if I can find a split that makes sense to me (without looking closely, I have a feeling that some of the steps you suggest might be too small, at least compared to others).

I'll try to answer your code-related comments tomorrow. I'm reasonably confident I'll eventually be able to actually address what should be done, though I won't make any promise on the timeline 😉.

[lanfeust69]

Not sure if a function only used with some features should be marked #[allow(dead_code)] (which may miss the time it is no longer used) or #[cfg(any(feature = "feature1", feature = "feature2"))] (which gets unwieldy, and may prevent discoverability if useful elsewhere)

We generally prefer the cfg-based approach, even though it can get unweildy, because it provides better compile times and code sizes. Discoverability is not an issue because we build docs for docs.rs with feature(doc_auto_cfg) which automatically adds "this method depends on blah feature" to all cfged functions.

Well, clippy barked at the allow(dead_code) version, so I ended up using the cfg-based version.

[alexcrichton]

Personally I'd prefer to split this into separate PRs if you're ok with that because there's a fair amount to discuss here and it's easy to lose track of discussions when it's all in one PR. I agree this is an unfortunate artifact of using github, but there's not a whole lot we can do about that other than work around it.

[vlhomutov]

I've tried to use the C Api as a user, and here is a couple of comments: do we really need wasmtime_component_val_t at all ? Conversion from wasmtime_val_t should be trivial and can be made somewhere inside the interface, if needed.

The second note is that wasmtime_component_instance_get_func() cannot find function if it is exported from interface, e.g.

package my:pkg;

interface my-stuff {
    my-func2: func(arg:s32) -> s32;
}

world my-world {
    export my-stuff;
    export my-func: func(arg:s32) -> s32;
}

when using module, my-func2 is accessible as my:pkg/my-stuff#my-func2. but with components, it cannot be found (using any name)

I checked the resulting wasm and see that component exports interface instance for my-stuff, and my-func is exported directly.

So I guess that either function search should be (maybe) recursive, or we need a way to traverse component contents.

Thank you for your effort, waiting for this to be finalized and accepted !

[lanfeust69]

Hi @vlhomutov,

Sorry I've had little time these past three months, and I missed your comment. It's a bit better now, so hopefully I'll be able to make progress on this PR (or rather on smaller ones after splitting this one).

Regarding the need of wasmtime_component_val_t, I think it is indeed necessary. Yes, for primitive types wasmtime_val_t is enough, but for all the rich structured types that the component model provides, you need more.

For wasmtime_component_instance_get_func(), it is probably related to the same kind of issues as the nesting of LinkerInstances, where the recursive patterns are somewhat hidden, but this may limit what can be expressed.

[MangoPeachGrape]

Hi @lanfeust69,

I would happily work on upstreaming this in smaller chunks, would that be okay with you? I could at least try to get the ball rolling, by doing "compile a component", as @alexcrichton suggested.

Thanks.

[lanfeust69]

Hi @MangoPeachGrape,

I could start taking a bit of time on this, including splitting the headers, which is a natural step for smaller PRs. But I'm not there yet...

[MangoPeachGrape]

Hi @lanfeust69,

I went ahead and split out the "compile a component" part into a seperate PR #10566, as an attempt to help upstream this, as I've been eagerly awaiting this feature for usage in my own project.

Feel free to let me know if you have any issues regarding this PR. Thanks.

[cyraid]

I'm also eagerly awaiting for this for a project too. You heroes got this.