Implement `wit pull`

[lann]

This implements a wit pull subcommand which can be used to update local e.g. wit/deps/*.wit from a registry.

$ wit pull wasi:io --create-dirs
   Resolving package wasi:io
  Downloaded release wasi:[email protected]
       Wrote package wasi:[email protected] to 'wit/deps/[email protected]'

$ head -n1 wit/deps/[email protected]
package wasi:[email protected];

$ wit pull wasi:http
   Resolving package wasi:http
  Downloaded release wasi:[email protected]
       Wrote package wasi:[email protected] to 'wit/deps/[email protected]'
       Wrote package wasi:[email protected] to 'wit/deps/[email protected]'
       Wrote package wasi:[email protected] to 'wit/deps/[email protected]'
       Wrote package wasi:[email protected] to 'wit/deps/[email protected]'

There are two modes of operation:

• When given specific name(s) it will pull those registry package(s) and unpack all (not existing) WIT packages into wit/deps/<namespace>-<name>@<version>.wit
• When given no name(s) it will check the packages under wit/ for missing dependencies and attempt to pull them like above

Both of these can operate whether or not wit_parser::Resolve::push_dir can currently parse the wit/ directory. If it can parse, the existing dependency checks will just be more accurate (capturing e.g. wit/deps/some-package.wasm). This ought to be more consistent but really needs some refactoring in wit-parser first.

[lann]

There is still much room for improvement but I think its useful enough as-is to iterate on.

[lann]

One open question is what to do with the warg-loader crate. At this point it could make sense to merge that into this repo.

[lann]

I am getting errors when trying this out myself, presumably because I don't have access to the bytecodealliance.org registry.

The BA registry should be public, e.g. https://github.com/orgs/bytecodealliance/packages/container/package/warg%2Fwasi%2Fio

[rylev]

This is the error I'm seeing when running the command against a wit world that uses a wasi:http dependency.

error: OCI error: Authentication failure: {"errors":[{"code":"DENIED","message":"denied"}]}


Caused by:
    Authentication failure: {"errors":[{"code":"DENIED","message":"denied"}]}

[rylev]

When I remove ghcr.io credentials by running docker logout ghcr.io then I end up with a different error:

error: failed to get registry credentials: Credential helper returned non-zero response code:
stdout:
credentials not found in native keychain


stderr:

And if I log back in with docker login ghcr.io then everything works as expected.

So it seems we're not allowing anonymous access to the ghcr instance and we're not handling the error case of when the credentials are bad.

I see you've opened https://github.com/lann/warg-loader/issues/5 as well which should hopefully address this.

[lann]

I made 2 changes in warg-loader which should fix the above auth issues:

• Tweaked auth helper error handling to treat a non-zero exit code as "no credential found"; this is apparently what is expected :shrug:
• Retry anonymous auth after credential failure, to catch the case where an expired token is used with a public registry

[lann]

Not sure how I'd test an expired token

I tested by corrupting a token in ~/.docker/config.json which seems to be the same error path as expiration.

[Zelzahn]

If I'm understanding it correctly, this will not serve as a replacement for wit-deps?

[lann]

If I'm understanding it correctly, this will not serve as a replacement for wit-deps?

It might eventually. Part of the purpose of this tool is to be a proving ground for some of the more advanced import definition types now available in the component model which should be able to replace external dependency config files.

See "options for naming imports" here: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#import-and-export-definitions

[rylev]

I believe this PR was closed by mistake by #270. I believe the correct issue to close was #268.

@peterhuene can you reopen this PR?

[peterhuene]

Whoops, sorry about that!

[calvinrp]

Closing this PR. The wit CLI functionality is in the process of moving over to https://github.com/bytecodealliance/wasm-pkg-tools